You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Khurram Faraaz (JIRA)" <ji...@apache.org> on 2016/03/16 19:42:33 UTC
[jira] [Commented] (DRILL-4330) Long running SQL query hangs once
Foreman node is killed
[ https://issues.apache.org/jira/browse/DRILL-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197904#comment-15197904 ]
Khurram Faraaz commented on DRILL-4330:
---------------------------------------
Problem is reproducible on Drill 1.6.0, JDK 7 and git commit ID : 64ab0a8ec9d98bf96f4d69274dddc180b8efe263
> Long running SQL query hangs once Foreman node is killed
> --------------------------------------------------------
>
> Key: DRILL-4330
> URL: https://issues.apache.org/jira/browse/DRILL-4330
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
> Affects Versions: 1.4.0
> Environment: 4 node cluster CentOS
> Reporter: Khurram Faraaz
> Assignee: Sudheesh Katkam
> Attachments: drillbit.out
>
>
> Summary : Once Foreman node Drillbit is killed, long running query just hangs and no profile information is written to Web UI. That long running query was issued from the Foreman node.
> MapR Drill 1.4.0 GA
> MapR FS 5.0.0 GA
> JDK8
> 4 node CentOS cluster
> ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
> Issue a long running select query over JSON data
> Immediately kill the Drillbit on Foreman node (ps -eaf | grep Drillbit), kill -9 PID
> The long running query hangs on sqlline prompt, there are no messages/errors/Exceptions reported on sqlline prompt.
> On the Web UI there is no profile reported for the long running query that was running on the Drillbit that was killed.
> Question (1) : Why was there no profile reported/written on the Web UI for that long running query ? In a real production scenario user will not know what query was under execution at the point when Foreman went down.
> Question (2) : Why does the long running query not terminate, once the foreman was killed ? from the drillbit.log snippet we do not see any CANCELED/TERMINATED message for that query, why ?
> Snippet from drillbit.log on the foreman node.
> {noformat}
> 2016-02-01 10:59:20,917 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 2950c576-b2d2-5bc3-e9b5-ff4414d088c0: select * from `twoKeyJsn.json`
> 2016-02-01 10:59:21,067 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.create() took 1 ms
> 2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,155 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Time: 90ms total, 90.891938ms avg, 90ms max.
> 2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Earliest start: 18.280000 μs, Latest start: 18.280000 μs, Average start: 18.280000 μs .
> 2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State change requested AWAITING_ALLOCATION --> RUNNING
> 2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State to report: RUNNING
> {noformat}
> Doing kill -3 PID on the non foreman node for the Drillbit process gives us stack trace in drillbit.out
> {noformat}
> 2016-02-01 11:03:31
> Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):
> "qtp801808302-129" #129 prio=5 os_prio=0 tid=0x00007f7ad8127000 nid=0xaad waiting on condition [0x00007f7ab62fb000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007b69100a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:513)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:569)
> at java.lang.Thread.run(Thread.java:745)
> "DestroyJavaVM" #84 prio=5 os_prio=0 tid=0x00007f7b20012000 nid=0x74c0 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "qtp801808302-81-acceptor-1-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #81 prio=5 os_prio=0 tid=0x00007f7b233e7000 nid=0x7a7d waiting for monitor entry [0x00007f7ab63fc000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
> - waiting to lock <0x00000007b1283570> (a java.lang.Object)
> at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-80-acceptor-0-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #80 prio=5 os_prio=0 tid=0x00007f7ad86df000 nid=0x7a7c runnable [0x00007f7ab64fd000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> - locked <0x00000007b1283570> (a java.lang.Object)
> at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-79-selector-ServerConnectorManager@6591a1ee/23" #79 prio=5 os_prio=0 tid=0x00007f7ad1368800 nid=0x7a7b runnable [0x00007f7ab65fe000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007b13f70d0> (a sun.nio.ch.Util$2)
> - locked <0x00000007b13f70c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000007b13f6fa8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
> at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-78-selector-ServerConnectorManager@6591a1ee/22" #78 prio=5 os_prio=0 tid=0x00000000026c2800 nid=0x7a7a runnable [0x00007f7ab66ff000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007b13f62e0> (a sun.nio.ch.Util$2)
> - locked <0x00000007b13f62d0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000007b13f61b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
> at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-77-selector-ServerConnectorManager@6591a1ee/21" #77 prio=5 os_prio=0 tid=0x00007f7b233e5000 nid=0x7a79 runnable [0x00007f7ab6800000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007b13f54f0> (a sun.nio.ch.Util$2)
> - locked <0x00000007b13f54e0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000007b13f53c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
> at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-76-selector-ServerConnectorManager@6591a1ee/20" #76 prio=5 os_prio=0 tid=0x00007f7ad86dd000 nid=0x7a78 runnable [0x00007f7ab6901000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007b13f4700> (a sun.nio.ch.Util$2)
> - locked <0x00000007b13f46f0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000007b13f45d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
> at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-74-selector-ServerConnectorManager@6591a1ee/19" #74 prio=5 os_prio=0 tid=0x00007f7ad13cb800 nid=0x7a77 runnable [0x00007f7ab6a01000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007b13f3910> (a sun.nio.ch.Util$2)
> - locked <0x00000007b13f3900> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000007b13f37e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
> at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-73-selector-ServerConnectorManager@6591a1ee/18" #73 prio=5 os_prio=0 tid=0x00000000026c1000 nid=0x7a76 runnable [0x00007f7ab6b03000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007b13f2b20> (a sun.nio.ch.Util$2)
> - locked <0x00000007b13f2b10> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000007b13f29f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
> at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-72-selector-ServerConnectorManager@6591a1ee/17" #72 prio=5 os_prio=0 tid=0x00007f7b233e3800 nid=0x7a75 runnable [0x00007f7ab6c04000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007b13f1ce0> (a sun.nio.ch.Util$2)
> - locked <0x00000007b13f1cd0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000007b13f1bb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
> at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-71-selector-ServerConnectorManager@6591a1ee/16" #71 prio=5 os_prio=0 tid=0x00007f7ad86db000 nid=0x7a74 runnable [0x00007f7ab6d05000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007b13e14e0> (a sun.nio.ch.Util$2)
> - locked <0x00000007b13e14d0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000007b13e13b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
> at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> ...
> "Scheduler-897230287" #62 prio=5 os_prio=0 tid=0x00007f7b23302800 nid=0x7a50 waiting on condition [0x00007f7ab770d000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007bb901728> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-61-selector-ServerConnectorManager@6591a1ee/7" #61 prio=5 os_prio=0 tid=0x00007f7b233b8000 nid=0x7a4f runnable [0x00007f7ab780e000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007b12fb2a0> (a sun.nio.ch.Util$2)
> - locked <0x00000007b12fb290> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000007b12fb178> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
> at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
> at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> ...
> "Curator-PathChildrenCache-1" #52 daemon prio=5 os_prio=0 tid=0x00007f7b2308c000 nid=0x7a47 waiting on condition [0x00007f7ab9179000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007bde07d08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "WorkManager.StatusThread" #40 daemon prio=5 os_prio=0 tid=0x00007f7b229af000 nid=0x7a45 waiting on condition [0x00007f7ab825a000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.drill.exec.work.WorkManager$StatusThread.run(WorkManager.java:328)
> "Curator-PathChildrenCache-0" #51 daemon prio=5 os_prio=0 tid=0x00007f7b2170f800 nid=0x79b2 waiting on condition [0x00007f7ab927a000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007b6b98818> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "BitServer-2" #21 daemon prio=10 os_prio=0 tid=0x00007f7b219d3800 nid=0x7981 runnable [0x00007f7ab940d000]
> java.lang.Thread.State: RUNNABLE
> at io.netty.channel.epoll.Native.epollWait0(Native Method)
> at io.netty.channel.epoll.Native.epollWait(Native.java:148)
> at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> "UserServer-1" #41 daemon prio=10 os_prio=0 tid=0x00007f7b21846800 nid=0x797f runnable [0x00007f7abb1a6000]
> java.lang.Thread.State: RUNNABLE
> at io.netty.channel.epoll.Native.epollWait0(Native Method)
> at io.netty.channel.epoll.Native.epollWait(Native.java:148)
> at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> "Curator-ServiceCache-0" #49 daemon prio=5 os_prio=0 tid=0x00007f7b2184d800 nid=0x796e waiting on condition [0x00007f7abc95d000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007b68066c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "Curator-Framework-0" #45 daemon prio=5 os_prio=0 tid=0x00007f7b21064800 nid=0x77ef waiting on condition [0x00007f7aba9a6000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007b6806b90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
> at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
> at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
> at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "main-EventThread" #44 daemon prio=5 os_prio=0 tid=0x00007f7b21043000 nid=0x77ee waiting on condition [0x00007f7abaaa7000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007b6806e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
> "main-SendThread(centos-01.qa.lab:5181)" #43 daemon prio=5 os_prio=0 tid=0x00007f7b20ec9000 nid=0x77ed runnable [0x00007f7abb02d000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007b5b00220> (a sun.nio.ch.Util$2)
> - locked <0x00000007b5b00230> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000007b5b001d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1070)
> "Curator-ConnectionStateManager-0" #42 daemon prio=5 os_prio=0 tid=0x00007f7b2223d000 nid=0x77ec waiting on condition [0x00007f7ababa8000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007b6807088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
> at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
> at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> ...
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f7b20cf8000 nid=0x7529 in Object.wait() [0x00007f7ac46d5000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000007b6807f58> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f7b20cf0000 nid=0x7528 in Object.wait() [0x00007f7ac47d6000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x00000007b5b00d88> (a java.lang.ref.Reference$Lock)
> ...
> Heap
> garbage-first heap total 4194304K, used 293887K [0x00000006c0000000, 0x00000006c0204000, 0x00000007c0000000)
> region size 2048K, 145 young (296960K), 21 survivors (43008K)
> Metaspace used 43409K, capacity 43751K, committed 44284K, reserved 1087488K
> class space used 5383K, capacity 5510K, committed 5628K, reserved 1048576K
> 2016-02-01 11:04:08
> Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):
> "qtp801808302-130" #130 prio=5 os_prio=0 tid=0x00007f7ad8125000 nid=0xc5b waiting on condition [0x00007f7ab61fa000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007b69100a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:513)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:569)
> at java.lang.Thread.run(Thread.java:745)
> "DestroyJavaVM" #84 prio=5 os_prio=0 tid=0x00007f7b20012000 nid=0x74c0 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "qtp801808302-81-acceptor-1-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #81 prio=5 os_prio=0 tid=0x00007f7b233e7000 nid=0x7a7d waiting for monitor entry [0x00007f7ab63fc000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
> - waiting to lock <0x00000007b1283570> (a java.lang.Object)
> at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)