You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jon Chase <jo...@gmail.com> on 2015/03/26 22:02:06 UTC

Spark SQL queries hang forever

Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB),
executor memory 20GB, driver memory 10GB

I'm using Spark SQL, mainly via spark-shell, to query 15GB of data spread
out over roughly 2,000 Parquet files and my queries frequently hang. Simple
queries like "select count(*) from ..." on the entire data set work ok.
Slightly more demanding ones with group by's and some aggregate functions
(percentile_approx, avg, etc.) work ok as well, as long as I have some
criteria in my where clause to keep the number of rows down.

Once I hit some limit on query complexity and rows processed, my queries
start to hang.  I've left them for up to an hour without seeing any
progress.  No OOM's either - the job is just stuck.

I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but
with the same results: usually near the end of the tasks (like 780 of 800
complete), progress just stops:

15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in
stage 1.0 (TID 1618) in 800 ms on
ip-10-209-22-211.eu-west-1.compute.internal (748/800)
15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in
stage 1.0 (TID 1623) in 622 ms on
ip-10-105-12-41.eu-west-1.compute.internal (749/800)
15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in
stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu-west-1.compute.internal
(750/800)
15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in
stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu-west-1.compute.internal
(751/800)
15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in
stage 1.0 (TID 1625) in 669 ms on
ip-10-105-12-41.eu-west-1.compute.internal (752/800)

^^^^^^^ this is where it stays forever

Looking at the Spark UI, several of the executors still list active tasks.
I do see that the Shuffle Read for executors that don't have any tasks
remaining is around 100MB, whereas it's more like 10MB for the executors
that still have tasks.

The first stage, mapPartitions, always completes fine.  It's the second
stage (takeOrdered), that hangs.

I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
encountered it when using JSON files (instead of Parquet).

Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do are
having this issue.

Re: Spark SQL queries hang forever

Posted by Jon Chase <jo...@gmail.com>.
I have experienced this indefinite hang when using sc.textFile() on JSON
files in addition to issues with Parquet, so it may not be file format
dependent.

I continue to hit this issue.  I wonder if the combination of Amazon
EMR/YARN, Spark 1.3 (also saw it on 1.2, 1.2.1), and Java8 could have any
affect.

The current job I'm working on runs fine with data sets, but hangs
indefinitely on larger ones.  It makes it through mapPartitions and then
gets stuck on a sc.union
(org.apache.spark.api.java.JavaSparkContextVarargsWorkaround.union(JavaSparkContextVarargsWorkaround.java:56)).
This has been stuck for about 24 hours now, and I just noticed that the
driver had a BLOCKED thread in the thread dump:


"qtp373544591-45 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #45 daemon
prio=5 os_prio=0 tid=0x00007f5b151f3800 nid=0x7600 waiting for monitor
entry [0x7600]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:233)
- waiting to lock <0x00000005403f3d68> (a java.lang.Object)
at
org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)




Here's the full driver thread dump if it's helpful:


Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):

"qtp373544591-2724" #2724 daemon prio=5 os_prio=0 tid=0x00007f5b00e05000
nid=0x5399 waiting on condition [0x00007f5ad9c41000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540426d20> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp373544591-2723" #2723 daemon prio=5 os_prio=0 tid=0x00007f5b00b9f000
nid=0x538d waiting on condition [0x00007f5aca702000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540426d20> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"IPC Parameter Sending Thread #1" #2254 daemon prio=5 os_prio=0
tid=0x00007f5b15f23800 nid=0x6392 waiting on condition [0x00007f5aca601000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000054040f660> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"sparkDriver-akka.actor.default-dispatcher-17" #1930 daemon prio=5
os_prio=0 tid=0x00007f5b15c2a000 nid=0x1d40 waiting on condition
[0x00007f5aca904000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540014298> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"ForkJoinPool-3-worker-15" #1093 daemon prio=5 os_prio=0
tid=0x00007f5b000af800 nid=0x586e waiting on condition [0x00007f5aca803000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000054e3b2ca8> (a
scala.concurrent.forkjoin.ForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"dag-scheduler-message-0" #1023 daemon prio=5 os_prio=0
tid=0x00007f5b006dc000 nid=0x4a2a waiting on condition [0x00007f5acac05000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000054037dcf0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"sparkDriver-akka.actor.default-dispatcher-16" #166 daemon prio=5 os_prio=0
tid=0x00007f5b16430800 nid=0x441 waiting on condition [0x00007f5acdd09000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540014298> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"shuffle-server-7" #79 daemon prio=5 os_prio=0 tid=0x00007f5af8093800
nid=0x3e9 runnable [0x00007f5ad4777000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000540669520> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005406b6ce8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000540669488> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-6" #78 daemon prio=5 os_prio=0 tid=0x00007f5af8833800
nid=0x3e8 runnable [0x00007f5adae6c000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005406b84a0> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005406ba520> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005406b8408> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"sparkDriver-akka.actor.default-dispatcher-15" #127 daemon prio=5 os_prio=0
tid=0x00007f5b014ed800 nid=0x7b81 waiting on condition [0x00007f5acde0a000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540014298> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"task-result-getter-3" #126 daemon prio=5 os_prio=0 tid=0x00007f5b0096e000
nid=0x7b80 waiting on condition [0x00007f5adac6a000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000054037e668> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"task-result-getter-2" #125 daemon prio=5 os_prio=0 tid=0x00007f5b00f58800
nid=0x7b7f waiting on condition [0x00007f5acdf0b000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000054037e668> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"task-result-getter-1" #124 daemon prio=5 os_prio=0 tid=0x00007f5b00b1c000
nid=0x7b7e waiting on condition [0x00007f5ace00c000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000054037e668> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"task-result-getter-0" #123 daemon prio=5 os_prio=0 tid=0x00007f5b00b1b000
nid=0x7b7d waiting on condition [0x00007f5ace10d000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000054037e668> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-5" #77 daemon prio=5 os_prio=0 tid=0x00007f5af8680000
nid=0x7b78 runnable [0x00007f5acf20f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005406bbcd8> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005406c7320> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005406bbc40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-4" #76 daemon prio=5 os_prio=0 tid=0x00007f5af87ce800
nid=0x7b77 runnable [0x00007f5ad4676000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005406c8ad8> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005406cab58> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005406c8a40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-3" #75 daemon prio=5 os_prio=0 tid=0x00007f5af8692800
nid=0x7b75 runnable [0x00007f5ad1312000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005406cc310> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005406d5d70> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005406cc278> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-2" #74 daemon prio=5 os_prio=0 tid=0x00007f5af8bb3000
nid=0x7b74 runnable [0x00007f5ad2414000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005406d7528> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005406d95a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005406d7490> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-2-1" #122 daemon prio=1 os_prio=0
tid=0x00007f5b150f7000 nid=0x7b71 waiting on condition [0x00007f5ad3716000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-1" #73 daemon prio=5 os_prio=0 tid=0x00007f5af8bb4000
nid=0x7b70 runnable [0x00007f5ad3817000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005406dad60> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005406dea10> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005406dacc8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"qtp593103894-121" #121 daemon prio=5 os_prio=0 tid=0x00007f5b1546f000
nid=0x7b6f waiting on condition [0x00007f5ad3918000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540355698> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp593103894-119" #119 daemon prio=5 os_prio=0 tid=0x00007f5b16248800
nid=0x7b6d waiting on condition [0x00007f5ad4878000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540355698> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"Attach Listener" #110 daemon prio=9 os_prio=0 tid=0x00007f5b15cdc800
nid=0x79fe waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"java-sdk-http-connection-reaper" #99 daemon prio=5 os_prio=0
tid=0x00007f5b0059b800 nid=0x764e waiting on condition [0x00007f5ad4979000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:112)

"com.google.inject.internal.util.$Finalizer" #94 daemon prio=5 os_prio=0
tid=0x00007f5b0027b000 nid=0x7648 in Object.wait() [0x00007f5ad4c7a000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000544989c30> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)

"BoneCP-pool-watch-thread" #92 daemon prio=5 os_prio=0
tid=0x00007f5b15cf7000 nid=0x7640 waiting on condition [0x00007f5ad65a7000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000544dfe2e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"BoneCP-keep-alive-scheduler" #91 daemon prio=5 os_prio=0
tid=0x00007f5b15cf5800 nid=0x763f waiting on condition [0x00007f5ad66a8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000544e01490> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"com.google.common.base.internal.Finalizer" #90 daemon prio=5 os_prio=0
tid=0x00007f5b15f78800 nid=0x763e in Object.wait() [0x00007f5ad67a9000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000544dff130> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
at java.lang.Thread.run(Thread.java:745)

"BoneCP-pool-watch-thread" #89 daemon prio=5 os_prio=0
tid=0x00007f5b15f82800 nid=0x7639 waiting on condition [0x00007f5ad68aa000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005449a4aa0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"BoneCP-keep-alive-scheduler" #88 daemon prio=5 os_prio=0
tid=0x00007f5b15f83800 nid=0x7638 waiting on condition [0x00007f5ad69ab000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005449a5a98> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"com.google.common.base.internal.Finalizer" #87 daemon prio=5 os_prio=0
tid=0x00007f5b15ee9000 nid=0x7637 in Object.wait() [0x00007f5ad6aac000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000005449a4910> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
at java.lang.Thread.run(Thread.java:745)

"Abandoned connection cleanup thread" #86 daemon prio=5 os_prio=0
tid=0x00007f5b15e19800 nid=0x7634 in Object.wait() [0x00007f5ad6fad000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000544d382f8> (a java.lang.ref.ReferenceQueue$Lock)
at
com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)

"sparkDriver-akka.actor.default-dispatcher-14" #85 daemon prio=5 os_prio=0
tid=0x00007f5b00390800 nid=0x7632 waiting on condition [0x00007f5ad8db7000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540014298> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"ResponseProcessor for block
BP-747680419-10.89.2.32-1428336502966:blk_1073741913_1089" #84 daemon
prio=5 os_prio=0 tid=0x00000000025b1000 nid=0x7623 runnable
[0x00007f5ad7cae000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005407d1168> (a sun.nio.ch.Util$2)
- locked <0x00000005407d1158> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005407d0f30> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)

"SparkListenerBus" #12 daemon prio=5 os_prio=0 tid=0x00007f5b15665000
nid=0x7622 waiting on condition [0x00007f5ad7daf000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005401c35f8> (a
java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)

"Spark Context Cleaner" #83 daemon prio=5 os_prio=0 tid=0x00007f5b15668000
nid=0x7621 in Object.wait() [0x00007f5ad7eb0000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000540668500> (a java.lang.ref.ReferenceQueue$Lock)
at
org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
at
org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
at
org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
at org.apache.spark.ContextCleaner.org
$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)

"DataStreamer for file
/spark-logs/application_1428336725156_0004.inprogress block
BP-747680419-10.89.2.32-1428336502966:blk_1073741913_1089" #82 daemon
prio=5 os_prio=0 tid=0x00007f5b15342800 nid=0x7620 in Object.wait()
[0x00007f5ad7fb1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
- locked <0x0000000540555020> (a java.util.LinkedList)

"metrics-ganglia-reporter-1-thread-1" #81 daemon prio=5 os_prio=0
tid=0x00007f5b15514800 nid=0x761f waiting on condition [0x00007f5ad80b2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000054058d3a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-0" #72 daemon prio=5 os_prio=0 tid=0x00007f5b15704000
nid=0x761e runnable [0x00007f5ad88b4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005406e0018> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005407162a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005406dff70> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"Yarn application state monitor" #63 daemon prio=5 os_prio=0
tid=0x00007f5b153b6000 nid=0x761b waiting on condition [0x00007f5ad8cb6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)

"Thread-32" #61 daemon prio=5 os_prio=0 tid=0x00007f5b15aad800 nid=0x7612
waiting on condition [0x00007f5ad87b3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005407189f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"LeaseRenewer:hadoop@10.89.2.32:9000" #59 daemon prio=5 os_prio=0
tid=0x00007f5b15a10800 nid=0x760f waiting on condition [0x00007f5ad8bb5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
at java.lang.Thread.run(Thread.java:745)

"Thread-26" #55 daemon prio=5 os_prio=0 tid=0x00007f5b159ce800 nid=0x760c
runnable [0x00007f5ad8eb8000]
   java.lang.Thread.State: RUNNABLE
at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
at
org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
at
org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
at java.lang.Thread.run(Thread.java:745)

"IPC Client (1381857776) connection to /10.89.2.32:9022 from hadoop" #53
daemon prio=5 os_prio=0 tid=0x00007f5b15864800 nid=0x760a in Object.wait()
[0x00007f5ad973e000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
- locked <0x00000005403ed498> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)

"dag-scheduler-event-loop" #52 daemon prio=5 os_prio=0
tid=0x00007f5b152dc800 nid=0x7609 waiting on condition [0x00007f5ad983f000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000054037df70> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
at
java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)

"Timer-0" #51 daemon prio=5 os_prio=0 tid=0x00007f5b1523a800 nid=0x7608 in
Object.wait() [0x00007f5ad9940000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x0000000540351738> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"qtp373544591-49" #49 daemon prio=5 os_prio=0 tid=0x00007f5b151fa800
nid=0x7604 waiting on condition [0x00007f5ad9d42000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540426d20> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp373544591-48" #48 daemon prio=5 os_prio=0 tid=0x00007f5b151f9000
nid=0x7603 waiting on condition [0x00007f5ad9e43000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540426d20> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp373544591-47" #47 daemon prio=5 os_prio=0 tid=0x00007f5b151f7000
nid=0x7602 waiting on condition [0x00007f5ad9f44000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540426d20> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp373544591-46 Acceptor1 SelectChannelConnector@0.0.0.0:4040" #46 daemon
prio=5 os_prio=0 tid=0x00007f5b151f5800 nid=0x7601 runnable
[0x00007f5ada045000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000005403f3d68> (a java.lang.Object)
at
org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

"qtp373544591-45 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #45 daemon
prio=5 os_prio=0 tid=0x00007f5b151f3800 nid=0x7600 waiting for monitor
entry [0x7600]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:233)
- waiting to lock <0x00000005403f3d68> (a java.lang.Object)
at
org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

"qtp373544591-44 Selector1" #44 daemon prio=5 os_prio=0
tid=0x00007f5b151f2000 nid=0x75ff runnable [0x00007f5ada247000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000540428848> (a sun.nio.ch.Util$2)
- locked <0x0000000540428838> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000540428710> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

"qtp373544591-43 Selector0" #43 daemon prio=5 os_prio=0
tid=0x00007f5b151f0800 nid=0x75fe runnable [0x00007f5ada348000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005404272d8> (a sun.nio.ch.Util$2)
- locked <0x00000005404272c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005404271a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

"SPARK_CONTEXT cleanup timer" #41 daemon prio=5 os_prio=0
tid=0x00007f5b15199000 nid=0x75fd in Object.wait() [0x00007f5ada866000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000540429e38> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x0000000540429e38> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"qtp593103894-40" #40 daemon prio=5 os_prio=0 tid=0x00007f5b15157800
nid=0x75fc waiting on condition [0x00007f5ada967000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540355698> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp593103894-39" #39 daemon prio=5 os_prio=0 tid=0x00007f5b15155800
nid=0x75fb waiting on condition [0x00007f5adaa68000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540355698> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp593103894-38" #38 daemon prio=5 os_prio=0 tid=0x00007f5b15153800
nid=0x75fa waiting on condition [0x00007f5adab69000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540355698> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp593103894-36" #36 daemon prio=5 os_prio=0 tid=0x00007f5b1514f800
nid=0x75f8 waiting on condition [0x00007f5adad6b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540355698> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp593103894-34" #34 daemon prio=5 os_prio=0 tid=0x00007f5b1514b800
nid=0x75f6 waiting on condition [0x00007f5adaf6d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540355698> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp593103894-33 Acceptor0 SocketConnector@0.0.0.0:44232" #33 daemon prio=5
os_prio=0 tid=0x00007f5b1514a800 nid=0x75f5 runnable [0x00007f5adb06e000]
   java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at
org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
at
org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

"BROADCAST_VARS cleanup timer" #31 daemon prio=5 os_prio=0
tid=0x00007f5b1505a800 nid=0x75f4 in Object.wait() [0x00007f5adb36f000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000540354648> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x0000000540354648> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"BLOCK_MANAGER cleanup timer" #30 daemon prio=5 os_prio=0
tid=0x00007f5b15054800 nid=0x75f3 in Object.wait() [0x00007f5adb470000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000540354128> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x0000000540354128> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"MAP_OUTPUT_TRACKER cleanup timer" #28 daemon prio=5 os_prio=0
tid=0x00007f5b1501b800 nid=0x75f2 in Object.wait() [0x00007f5adb571000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000540352288> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x0000000540352288> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"New I/O server boss #6" #27 daemon prio=5 os_prio=0 tid=0x00007f5af81c9000
nid=0x75f1 runnable [0x00007f5adb872000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005403daf80> (a sun.nio.ch.Util$2)
- locked <0x00000005403daf70> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005403dabb8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #5" #26 daemon prio=5 os_prio=0 tid=0x00007f5af81c8000
nid=0x75f0 runnable [0x00007f5adb973000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005403d92d0> (a sun.nio.ch.Util$2)
- locked <0x00000005403d92c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005403d91a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #4" #25 daemon prio=5 os_prio=0 tid=0x00007f5af8304000
nid=0x75ef runnable [0x00007f5adba74000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005403d7d08> (a sun.nio.ch.Util$2)
- locked <0x00000005403d7cf8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005403d7be0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O boss #3" #24 daemon prio=5 os_prio=0 tid=0x00007f5af818c000
nid=0x75ed runnable [0x00007f5adbb75000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000054042a3d0> (a sun.nio.ch.Util$2)
- locked <0x000000054042a3e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000054042a388> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #2" #22 daemon prio=5 os_prio=0 tid=0x00007f5af81cc800
nid=0x75eb runnable [0x00007f5adbc76000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000540460d40> (a sun.nio.ch.Util$2)
- locked <0x0000000540460d50> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000540460cf8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #1" #21 daemon prio=5 os_prio=0 tid=0x00007f5af81d1800
nid=0x75ea runnable [0x00007f5adbd77000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000054047fc18> (a sun.nio.ch.Util$2)
- locked <0x000000054047fc28> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000054047fbd0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"sparkDriver-akka.remote.default-remote-dispatcher-6" #20 daemon prio=5
os_prio=0 tid=0x00007f5b14fe7800 nid=0x75e9 waiting on condition
[0x00007f5adc089000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000054001c4f8> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-akka.remote.default-remote-dispatcher-5" #19 daemon prio=5
os_prio=0 tid=0x00007f5af8104000 nid=0x75e8 waiting on condition
[0x00007f5adc18a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000054001c4f8> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at
scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-akka.actor.default-dispatcher-4" #18 daemon prio=5 os_prio=0
tid=0x00007f5af81e6000 nid=0x75e7 waiting on condition [0x00007f5adc28b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540014298> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at
scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-akka.actor.default-dispatcher-3" #17 daemon prio=5 os_prio=0
tid=0x00007f5b14fba800 nid=0x75e6 waiting on condition [0x00007f5adc38c000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540014298> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-akka.actor.default-dispatcher-2" #16 daemon prio=5 os_prio=0
tid=0x00007f5b14fa9800 nid=0x75e5 waiting on condition [0x00007f5adc48d000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000540014298> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-scheduler-1" #15 daemon prio=5 os_prio=0
tid=0x00007f5b14f22000 nid=0x75e4 waiting on condition [0x00007f5adcb8f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
at
akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f5b140f3800
nid=0x75df runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f5b140ce800
nid=0x75de waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f5b140cc000
nid=0x75dd waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f5b140ca800
nid=0x75dc waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f5b140c7800
nid=0x75db waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f5b140c6000
nid=0x75da runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f5b1408e000 nid=0x75d9 in
Object.wait() [0x00007f5afd9ec000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000540037ab8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f5b1408c000
nid=0x75d8 in Object.wait() [0x00007f5afdaed000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x00000005400290f8> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x00007f5b14011000 nid=0x75ce in
Object.wait() [0x00007f5b1de0e000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
- locked <0x000000054502b3f0> (a org.apache.spark.scheduler.JobWaiter)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1483)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1504)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1087)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:954)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:916)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:874)
at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1308)
at
org.apache.spark.api.java.JavaRDDLike$class.saveAsTextFile(JavaRDDLike.scala:497)
at org.apache.spark.api.java.JavaRDD.saveAsTextFile(JavaRDD.scala:32)
at
com.cedexis.ads.reports.traffic_calculator.TrafficCalculator.saveReport(TrafficCalculator.java:138)
at
com.cedexis.ads.reports.traffic_calculator.TrafficCalculator.runJob(TrafficCalculator.java:186)
at
com.cedexis.ads.reports.traffic_calculator.TrafficCalculator.main(TrafficCalculator.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

"VM Thread" os_prio=0 tid=0x00007f5b14087000 nid=0x75d7 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f5b14026800 nid=0x75cf
runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f5b14028000 nid=0x75d0
runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f5b1402a000 nid=0x75d1
runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f5b1402b800 nid=0x75d2
runnable

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f5b1402d800 nid=0x75d3
runnable

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f5b1402f000 nid=0x75d4
runnable

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f5b14031000 nid=0x75d5
runnable

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f5b14032800 nid=0x75d6
runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f5b140f6000 nid=0x75e0
waiting on condition

JNI global references: 331




On Thu, Apr 2, 2015 at 3:32 PM, Christian Perez <ch...@svds.com> wrote:

> I found that rewriting our tables in Impala (originally Hive) solved
> the problem. The files are bigger so that there are 180 instead of
> 718.
>
> I also think that turning the shuffle.partitions down to 100 on the
> original set prevented Spark from hanging, but haven't tried to
> reproduce since finding this workaround.
>
> Perhaps there's an issue with either a) hidden Spark issues with the
> particular parquet files or b) subtle bugs that appear later when
> Spark processes many files. **A shuffle hang occurred with >800 text
> files of same data.**
>
> On Wed, Apr 1, 2015 at 6:39 AM, Marius Soutier <mp...@gmail.com> wrote:
> > I have similiar problem since upgrading to 1.2.1.
> >
> > SparkSQL queries on small, simple Parquet files take up to 10 minutes.
> After an initial hang, it seems to work fine for a while.
> >
> > The problematic stage is RangePartitioner at Exchange.scala:88.
> >
> >
> >> On 31.03.2015, at 21:55, Christian Perez <ch...@svds.com> wrote:
> >>
> >> I think I've seen a timeout in one of the executor logs... but I'm
> >> usually not that patient.
> >>
> >> If I kill the executor running the last task started on spark UI, 2
> >> out of 3 times it will retry and complete successfully.
> >>
> >> I've found that everything works when running in local mode versus
> >> YARN. Have not tried Spark standalone. Something (block fetcher?) is
> >> hanging during shuffle when distributed...
> >>
> >> On Tue, Mar 31, 2015 at 12:10 AM, Zsolt Tóth <to...@gmail.com>
> wrote:
> >>> I have a very similar problem with collect() but only on relatively
> large
> >>> input data, on Amazon EMR. Look for my thread for details if
> interested.
> >>> I get "connection timed out" exception after a 15-20 minutes in one of
> the
> >>> container logs - does it happen in your case?
> >>>
> >>> 2015-03-30 23:34 GMT+02:00 Christian Perez <ch...@svds.com>:
> >>>>
> >>>> I'm having the exact same problem (hanging) with takeOrdered (using
> >>>> PySpark's orderBy on a DataFrame.)
> >>>>
> >>>> On Thu, Mar 26, 2015 at 7:21 PM, Jon Chase <jo...@gmail.com>
> wrote:
> >>>>> Here are the jstacks from a 3 worker cluster.  Two of the workers are
> >>>>> sitting with 4 active tasks, the other has no active tasks.  Not
> sure if
> >>>>> it
> >>>>> makes a difference, but I only see an ExecutorLauncher instance via
> JPS
> >>>>> on
> >>>>> the worker that has no active tasks.
> >>>>>
> >>>>> Here's the first stack dump, this machine has 4 active tasks:
> >>>>>
> >>>>> 2015-03-27 02:11:46
> >>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> >>>>> mode):
> >>>>>
> >>>>> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
> >>>>> nid=0x43a3 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
> >>>>> os_prio=0
> >>>>> tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
> >>>>> [0x00007f91bad9a000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x0000000589884fc0> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
> >>>>> nid=0x42e2 runnable [0x00007f91b4c54000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000005898aa000> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x00000005898c4b80> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
> >>>>> nid=0x42e1 runnable [0x00007f91b5d56000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058ad5b5f8> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058ad5b380> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
> >>>>> nid=0x42e0 runnable [0x00007f91b5e57000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000005898aa0c0> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x00000005898dac48> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
> >>>>> nid=0x42df runnable [0x00007f91b5f58000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058ad5b6b8> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058ad6f698> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f91d0773800 nid=0x427b waiting on condition
> >>>>> [0x00007f91b6258000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000006cfda6800> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> >>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> >>>>> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5
> os_prio=0
> >>>>> tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition
> >>>>> [0x00007f91b6b5a000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> >>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> >>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000
> nid=0x41f2
> >>>>> waiting on condition [0x00007f91b6c5b000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x000000058af8e0b8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000
> nid=0x41ef
> >>>>> runnable [0x00007f91baf9c000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native
> Method)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0
> >>>>> tid=0x00007f91d8118800
> >>>>> nid=0x41ee waiting on condition [0x00007f91bae9b000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> >>>>>
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
> >>>>> nid=0x41ed runnable [0x00007f91baa97000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x0000000589224af0> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x00000005894b8d70> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f91d0019000 nid=0x41ea waiting on condition
> >>>>> [0x00007f91bac98000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000006c8837750> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> >>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> >>>>> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f91d0016800 nid=0x41e9 waiting on condition
> >>>>> [0x00007f91ba16d000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000006cc53bff0> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> >>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> >>>>> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f91d0015000 nid=0x41e8 waiting on condition
> >>>>> [0x00007f91b9f6b000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000006cfda69b0> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> >>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> >>>>> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Driver Heartbeater" #55 daemon prio=5 os_prio=0
> tid=0x000000000144e800
> >>>>> nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
> >>>>>
> >>>>> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
> >>>>> nid=0x41c6 runnable [0x00007f91bbac7000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058984ee60> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058984c368> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition
> >>>>> [0x00007f91bbbc8000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000005898f7290> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait()
> [0x00007f91bbdca000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
> >>>>> - locked <0x000000058974cb00> (a java.util.TaskQueue)
> >>>>> at java.util.TimerThread.run(Timer.java:505)
> >>>>>
> >>>>> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait()
> [0x00007f91bbcc9000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> - waiting on <0x0000000589904038> (a java.util.TaskQueue)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
> >>>>> - locked <0x0000000589904038> (a java.util.TaskQueue)
> >>>>> at java.util.TimerThread.run(Timer.java:505)
> >>>>>
> >>>>> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f91ed92d800
> >>>>> nid=0x41bf waiting on condition [0x00007f91bab98000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x0000000589884e80> (a
> >>>>> scala.concurrent.forkjoin.ForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f91ed92c800
> >>>>> nid=0x41be waiting on condition [0x00007f91ba370000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O server boss #12" #39 daemon prio=5 os_prio=0
> >>>>> tid=0x0000000000a02000
> >>>>> nid=0x41bd runnable [0x00007f91b9564000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x000000058992f170> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #11" #38 daemon prio=5 os_prio=0
> tid=0x0000000000a00000
> >>>>> nid=0x41bc runnable [0x00007f91b9665000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x0000000589931200> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #10" #37 daemon prio=5 os_prio=0
> tid=0x00000000009fe800
> >>>>> nid=0x41bb runnable [0x00007f91b9766000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x0000000589944b38> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
> >>>>> nid=0x41ba runnable [0x00007f91b9867000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x0000000589977fb0> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #8" #34 daemon prio=5 os_prio=0
> tid=0x0000000000cfe800
> >>>>> nid=0x41b9 runnable [0x00007f91b9968000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x000000058998e4c0> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #7" #33 daemon prio=5 os_prio=0
> tid=0x00000000009b5000
> >>>>> nid=0x41b8 runnable [0x00007f91b9c69000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x0000000589990bc8> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon
> >>>>> prio=5
> >>>>> os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
> >>>>> [0x00007f91b9d6a000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x0000000589884f20> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon
> >>>>> prio=5
> >>>>> os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
> >>>>> [0x00007f91b9e6b000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x0000000589884f20> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> >>>>> at
> >>>>>
> >>>>>
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> >>>>> at
> >>>>>
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
> >>>>> nid=0x419e runnable [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0
> tid=0x00007f91ec109000
> >>>>> nid=0x419d waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0
> tid=0x00007f91ec107000
> >>>>> nid=0x419c waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0
> tid=0x00007f91ec105000
> >>>>> nid=0x419b waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
> >>>>> nid=0x419a runnable [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> >>>>> tid=0x00007f91ec101000 nid=0x4199 waiting on condition
> >>>>> [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000
> nid=0x4198
> >>>>> in
> >>>>> Object.wait() [0x00007f91d4254000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> >>>>> - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> >>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> >>>>>
> >>>>> "Reference Handler" #2 daemon prio=10 os_prio=0
> tid=0x00007f91ec0c9800
> >>>>> nid=0x4197 in Object.wait() [0x00007f91d4355000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> >>>>> - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
> >>>>>
> >>>>> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting
> on
> >>>>> condition [0x00007f91f1fdf000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x0000000589a1b5f0> (a
> >>>>> java.util.concurrent.CountDownLatch$Sync)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> >>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> >>>>> at
> >>>>>
> >>>>>
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> >>>>> at
> >>>>>
> >>>>>
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> >>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> >>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> >>>>> at scala.concurrent.Await$.ready(package.scala:86)
> >>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> >>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> >>>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> >>>>>
> >>>>> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
> >>>>>
> >>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0
> tid=0x00007f91ec02a000
> >>>>> nid=0x4191 runnable
> >>>>>
> >>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0
> tid=0x00007f91ec02b800
> >>>>> nid=0x4192 runnable
> >>>>>
> >>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0
> tid=0x00007f91ec02d800
> >>>>> nid=0x4193 runnable
> >>>>>
> >>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0
> tid=0x00007f91ec02f000
> >>>>> nid=0x4194 runnable
> >>>>>
> >>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
> >>>>> nid=0x4195 runnable
> >>>>>
> >>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
> >>>>> waiting on condition
> >>>>>
> >>>>> JNI global references: 319
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Here's the next worker, also for active tasks:
> >>>>>
> >>>>> 2015-03-27 02:12:54
> >>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> >>>>> mode):
> >>>>>
> >>>>> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
> >>>>> nid=0x44f1 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
> >>>>> os_prio=0
> >>>>> tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
> >>>>> [0x00007fcbbf87f000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x0000000589510428> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fcbe4028800 nid=0x445f waiting on condition
> >>>>> [0x00007fcbbf97f000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000006126cd358> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> >>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> >>>>> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
> >>>>> nid=0x4459 runnable [0x00007fcbc88c7000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058a6b8c98> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058a6bd270> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
> >>>>> nid=0x4458 runnable [0x00007fcbc7897000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058a6e5a18> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058a6e4f60> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
> >>>>> nid=0x4457 runnable [0x00007fcbc7998000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058aade110> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058aadd508> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
> >>>>> nid=0x4456 runnable [0x00007fcbc7b9a000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058aade1d0> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058aaf16b8> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition
> >>>>> [0x00007fcbc4886000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x0000000615539dd8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> >>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> >>>>> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fcbe4165000 nid=0x4453 waiting on condition
> >>>>> [0x00007fcbca620000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x0000000617bd4238> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> >>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> >>>>> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fcbe4164000 nid=0x4452 waiting on condition
> >>>>> [0x00007fcbc87c5000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000006126cd538> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> >>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at
> >>>>>
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> >>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> >>>>> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5
> os_prio=0
> >>>>> tid=0x00007fcbdcff1800 nid=0x436e waiting on condition
> >>>>> [0x00007fcbc4485000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> >>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> >>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800
> nid=0x4369
> >>>>> waiting on condition [0x00007fcbc4586000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x000000058adabe70> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800
> nid=0x4366
> >>>>> runnable [0x00007fcbc7e9d000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native
> Method)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> >>>>> tid=0x00007fcbf9608800
> >>>>> nid=0x4365 waiting on condition [0x00007fcbc89c8000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> >>>>>
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
> >>>>> nid=0x4364 runnable [0x00007fcbc8ac9000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x0000000589116ae0> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x00000005892ef140> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0
> tid=0x0000000003ecc000
> >>>>> nid=0x4341 waiting on condition [0x00007fcbc95f4000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
> >>>>>
> >>>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
> >>>>> nid=0x4340 runnable [0x00007fcbc94f3000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058964c068> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058a691b28> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition
> >>>>> [0x00007fcbc97f6000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x000000058a6fa058> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait()
> [0x00007fcbc98f7000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> - waiting on <0x000000058928c438> (a java.util.TaskQueue)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
> >>>>> - locked <0x000000058928c438> (a java.util.TaskQueue)
> >>>>> at java.util.TimerThread.run(Timer.java:505)
> >>>>>
> >>>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fcbf9b49000 nid=0x433b in Object.wait()
> [0x00007fcbc96f5000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
> >>>>> - locked <0x000000058928c5a0> (a java.util.TaskQueue)
> >>>>> at java.util.TimerThread.run(Timer.java:505)
> >>>>>
> >>>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fcbdc29d000
> >>>>> nid=0x433a waiting on condition [0x00007fcbc85c4000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fcbe4474800
> >>>>> nid=0x4338 waiting on condition [0x00007fcbc86c5000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x000000058a70a468> (a
> >>>>> scala.concurrent.forkjoin.ForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fcbdc29a000
> >>>>> nid=0x4337 runnable [0x00007fcbc6e8f000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x000000058a724d50> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0
> tid=0x00007fcbdc299000
> >>>>> nid=0x4336 runnable [0x00007fcbc6f90000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x000000058a73ee00> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0
> tid=0x00007fcbdc356800
> >>>>> nid=0x4335 runnable [0x00007fcbc7091000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x000000058a73f198> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
> >>>>> nid=0x4334 runnable [0x00007fcbc7392000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x000000058a796538> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0
> tid=0x00007fcbdc298000
> >>>>> nid=0x4333 runnable [0x00007fcbc7493000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x000000058a7be828> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0
> tid=0x00007fcbdc257000
> >>>>> nid=0x4332 runnable [0x00007fcbc7594000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x000000058a7bed80> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon
> >>>>> prio=5
> >>>>> os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
> >>>>> [0x00007fcbc7695000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000005893c1820> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon
> >>>>> prio=5
> >>>>> os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
> >>>>> [0x00007fcbc7796000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000005893c1820> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fcbf995d800 nid=0x432a waiting on condition
> >>>>> [0x00007fcbc7d9c000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> >>>>> at
> >>>>>
> >>>>>
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> >>>>> at
> >>>>>
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
> >>>>> nid=0x4318 runnable [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0
> tid=0x00007fcbf810b800
> >>>>> nid=0x4317 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0
> tid=0x00007fcbf8109000
> >>>>> nid=0x4316 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0
> tid=0x00007fcbf8107000
> >>>>> nid=0x4315 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
> >>>>> nid=0x4314 runnable [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> >>>>> tid=0x00007fcbf8103800 nid=0x4313 waiting on condition
> >>>>> [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000
> nid=0x4312
> >>>>> in
> >>>>> Object.wait() [0x00007fcbe1be7000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> >>>>> - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> >>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> >>>>>
> >>>>> "Reference Handler" #2 daemon prio=10 os_prio=0
> tid=0x00007fcbf80c9800
> >>>>> nid=0x4311 in Object.wait() [0x00007fcbe8150000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> >>>>> - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
> >>>>>
> >>>>> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting
> on
> >>>>> condition [0x00007fcbff9b9000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x0000000589585398> (a
> >>>>> java.util.concurrent.CountDownLatch$Sync)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> >>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> >>>>> at
> >>>>>
> >>>>>
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> >>>>> at
> >>>>>
> >>>>>
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> >>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> >>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> >>>>> at scala.concurrent.Await$.ready(package.scala:86)
> >>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> >>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> >>>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> >>>>>
> >>>>> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
> >>>>>
> >>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0
> tid=0x00007fcbf802a000
> >>>>> nid=0x430b runnable
> >>>>>
> >>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0
> tid=0x00007fcbf802b800
> >>>>> nid=0x430c runnable
> >>>>>
> >>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0
> tid=0x00007fcbf802d800
> >>>>> nid=0x430d runnable
> >>>>>
> >>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0
> tid=0x00007fcbf802f000
> >>>>> nid=0x430e runnable
> >>>>>
> >>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
> >>>>> nid=0x430f runnable
> >>>>>
> >>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
> >>>>> waiting on condition
> >>>>>
> >>>>> JNI global references: 375
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> And the final worker, no active tasks:
> >>>>>
> >>>>>
> >>>>> 2015-03-27 02:13:26
> >>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> >>>>> mode):
> >>>>>
> >>>>> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
> >>>>> nid=0x3c5b waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
> >>>>> os_prio=0
> >>>>> tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
> >>>>> [0x00007f36d2e33000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000005895676e8> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
> >>>>> nid=0x3b88 runnable [0x00007f36dafb9000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058a86a7d8> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058a8818c8> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
> >>>>> nid=0x3b87 runnable [0x00007f36d7138000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058abea6c8> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058abea488> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
> >>>>> nid=0x3b86 runnable [0x00007f36daeb8000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058a86a898> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058a89a400> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
> >>>>> nid=0x3b85 runnable [0x00007f36d7e3b000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058abea788> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058abff150> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5
> os_prio=0
> >>>>> tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition
> >>>>> [0x00007f36d7a39000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> >>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> >>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000
> nid=0x3a9d
> >>>>> waiting on condition [0x00007f36d7b3a000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x000000058aecd008> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800
> nid=0x3a9a
> >>>>> runnable [0x00007f36db4be000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native
> Method)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> >>>>> tid=0x00007f36f8025000
> >>>>> nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> >>>>>
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
> >>>>> nid=0x3a96 runnable [0x00007f36dc0ea000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x0000000589202c58> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x0000000589390308> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0
> tid=0x00007f36f8586000
> >>>>> nid=0x3a6d waiting on condition [0x00007f36dcc15000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
> >>>>>
> >>>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
> >>>>> nid=0x3a6c runnable [0x00007f36dcb14000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058a825c18> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x000000058a827090> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition
> >>>>> [0x00007f36dce17000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000005897334d0> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f370da11800 nid=0x3a68 in Object.wait()
> [0x00007f36dcd16000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
> >>>>> - locked <0x00000005896b0d20> (a java.util.TaskQueue)
> >>>>> at java.util.TimerThread.run(Timer.java:505)
> >>>>>
> >>>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f370da10800 nid=0x3a67 in Object.wait()
> [0x00007f36dcf18000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
> >>>>> - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
> >>>>> at java.util.TimerThread.run(Timer.java:505)
> >>>>>
> >>>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f36f03a1000
> >>>>> nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f370d9b2000
> >>>>> nid=0x3a64 waiting on condition [0x00007f36dbde7000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000005895675a8> (a
> >>>>> scala.concurrent.forkjoin.ForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f36f8713000
> >>>>> nid=0x3a63 runnable [0x00007f36da4b0000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x000000058970f520> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0
> tid=0x00007f36f86e8800
> >>>>> nid=0x3a62 runnable [0x00007f36da5b1000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000005896e8698> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0
> tid=0x00007f36f86bc800
> >>>>> nid=0x3a61 runnable [0x00007f36da6b2000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000005896e8560> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
> >>>>> nid=0x3a60 runnable [0x00007f36da7b3000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x000000058969cec8> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0
> tid=0x00007f36f8676800
> >>>>> nid=0x3a5f runnable [0x00007f36da8b4000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000005896c8fd0> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0
> tid=0x00007f36f864c800
> >>>>> nid=0x3a5e runnable [0x00007f36da9b5000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000005896b3328> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon
> >>>>> prio=5
> >>>>> os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
> >>>>> [0x00007f36dacb6000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x0000000589567648> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon
> >>>>> prio=5
> >>>>> os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
> >>>>> [0x00007f36dadb7000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x0000000589567648> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> >>>>> tid=0x00007f370d97c000 nid=0x3a56 waiting on condition
> >>>>> [0x00007f36db3bd000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> >>>>> at
> >>>>>
> >>>>>
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> >>>>> at
> >>>>>
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
> >>>>> nid=0x3a43 runnable [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0
> tid=0x00007f370c109000
> >>>>> nid=0x3a42 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0
> tid=0x00007f370c107000
> >>>>> nid=0x3a41 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0
> tid=0x00007f370c105000
> >>>>> nid=0x3a40 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
> >>>>> nid=0x3a3f runnable [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> >>>>> tid=0x00007f370c101800 nid=0x3a3e waiting on condition
> >>>>> [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000
> nid=0x3a3d
> >>>>> in
> >>>>> Object.wait() [0x00007f36f51e5000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> >>>>> - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> >>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> >>>>>
> >>>>> "Reference Handler" #2 daemon prio=10 os_prio=0
> tid=0x00007f370c0c9800
> >>>>> nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> >>>>> - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
> >>>>>
> >>>>> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting
> on
> >>>>> condition [0x00007f3712fda000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x000000058a974ab8> (a
> >>>>> java.util.concurrent.CountDownLatch$Sync)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> >>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> >>>>> at
> >>>>>
> >>>>>
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> >>>>> at
> >>>>>
> >>>>>
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> >>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> >>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> >>>>> at scala.concurrent.Await$.ready(package.scala:86)
> >>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> >>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> >>>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> >>>>>
> >>>>> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
> >>>>>
> >>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0
> tid=0x00007f370c02a000
> >>>>> nid=0x3a34 runnable
> >>>>>
> >>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0
> tid=0x00007f370c02b800
> >>>>> nid=0x3a35 runnable
> >>>>>
> >>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0
> tid=0x00007f370c02d800
> >>>>> nid=0x3a36 runnable
> >>>>>
> >>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0
> tid=0x00007f370c02f000
> >>>>> nid=0x3a37 runnable
> >>>>>
> >>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
> >>>>> nid=0x3a3a runnable
> >>>>>
> >>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
> >>>>> waiting on condition
> >>>>>
> >>>>> JNI global references: 486
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> And the driver:
> >>>>>
> >>>>> 2015-03-27 02:20:47
> >>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> >>>>> mode):
> >>>>>
> >>>>> "qtp1576607881-152" #152 daemon prio=5 os_prio=0
> tid=0x00007fd3d400b800
> >>>>> nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e67194f0> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1576607881-147" #147 daemon prio=5 os_prio=0
> tid=0x00007fd3ec5e5000
> >>>>> nid=0x465a waiting on condition [0x00007fd3b8eb5000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e67194f0> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1576607881-145" #145 daemon prio=5 os_prio=0
> tid=0x00007fd3e8092800
> >>>>> nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e67194f0> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1576607881-122" #122 daemon prio=5 os_prio=0
> tid=0x00007fd3f401f000
> >>>>> nid=0x41d0 waiting on condition [0x00007fd38feef000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e67194f0> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition
> >>>>> [0x00007fd38fff0000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000fd545eb8> (a
> >>>>> scala.concurrent.forkjoin.ForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5
> >>>>> os_prio=0
> >>>>> tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition
> >>>>> [0x00007fd3b8400000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e64f49f8> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "qtp1576607881-115" #115 daemon prio=5 os_prio=0
> tid=0x00007fd3dc018000
> >>>>> nid=0x417f waiting on condition [0x00007fd398935000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e67194f0> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "task-result-getter-3" #107 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd3b003f800
> >>>>> nid=0x409c waiting on condition [0x00007fd3902f1000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e66e9a00> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "task-result-getter-2" #106 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd3b003d000
> >>>>> nid=0x409b waiting on condition [0x00007fd3903f2000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e66e9a00> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "task-result-getter-1" #105 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd3b003b800
> >>>>> nid=0x409a waiting on condition [0x00007fd3904f3000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e66e9a00> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "task-result-getter-0" #104 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd3b0030800
> >>>>> nid=0x4098 waiting on condition [0x00007fd3905f4000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e66e9a00> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
> >>>>> tid=0x00007fd3cc00c800
> >>>>> nid=0x4093 waiting on condition [0x00007fd3936f8000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> >>>>>
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
> >>>>> nid=0x4092 runnable [0x00007fd398b37000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e685f320> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x00000000e685f380> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
> >>>>> nid=0x4091 runnable [0x00007fd398a36000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e6871c30> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x00000000e6872ca0> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
> >>>>> nid=0x4090 runnable [0x00007fd39903a000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e6873520> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x00000000e6873580> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5
> os_prio=0
> >>>>> tid=0x00007fd3e0011800 nid=0x4056 waiting on condition
> >>>>> [0x00007fd398733000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> >>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> >>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd3dc001800 nid=0x4055 waiting on condition
> >>>>> [0x00007fd398834000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e7f19070> (a
> >>>>> scala.concurrent.forkjoin.ForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5
> >>>>> os_prio=0
> >>>>> tid=0x00007fd3a400d800 nid=0x4051 waiting on condition
> >>>>> [0x00007fd398e38000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e64f49f8> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd432c05000 nid=0x404f waiting on condition
> >>>>> [0x00007fd398f39000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e66fe220> (a
> >>>>> java.util.concurrent.SynchronousQueue$TransferStack)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd43297c800 nid=0x4042 waiting on condition
> >>>>> [0x00007fd39973b000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e740fa58> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> >>>>> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd4329ac000 nid=0x4041 waiting on condition
> >>>>> [0x00007fd39983c000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e740fce8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "com.google.common.base.internal.Finalizer" #91 daemon prio=5
> os_prio=0
> >>>>> tid=0x00007fd4329aa000 nid=0x4040 in Object.wait()
> [0x00007fd39993d000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> >>>>> - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> >>>>> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd4328e6800 nid=0x403b waiting on condition
> >>>>> [0x00007fd399c3e000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e737a0c8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> >>>>> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd4328eb800 nid=0x403a waiting on condition
> >>>>> [0x00007fd399d3f000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e737add8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "com.google.common.base.internal.Finalizer" #88 daemon prio=5
> os_prio=0
> >>>>> tid=0x00007fd432532000 nid=0x4039 in Object.wait()
> [0x00007fd399e40000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> >>>>> - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> >>>>> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd431b97000 nid=0x4036 in Object.wait()
> [0x00007fd39bdfe000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> >>>>> - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
> >>>>> at
> >>>>>
> >>>>>
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
> >>>>>
> >>>>> "ResponseProcessor for block
> >>>>> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86
> daemon
> >>>>> prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
> >>>>> [0x00007fd39a941000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000000e67d6038> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> >>>>> at
> >>>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> >>>>> at
> >>>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> >>>>> at
> >>>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> >>>>> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> >>>>> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
> >>>>>
> >>>>> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
> >>>>> nid=0x3fda waiting on condition [0x00007fd39aa42000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e58145f0> (a
> >>>>> java.util.concurrent.Semaphore$NonfairSync)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> >>>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> >>>>> at
> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
> >>>>>
> >>>>> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd43258c000
> >>>>> nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> >>>>> - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> >>>>> at
> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> >>>>> at
> >>>>>
> >>>>> org.apache.spark.ContextCleaner.org
> $apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
> >>>>> at
> org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
> >>>>>
> >>>>> "DataStreamer for file
> >>>>> /spark-logs/application_1427416187821_0004.inprogress
> >>>>> block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699"
> #84
> >>>>> daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in
> >>>>> Object.wait()
> >>>>> [0x00007fd39ae44000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
> >>>>> - locked <0x00000000e6994f30> (a java.util.LinkedList)
> >>>>>
> >>>>> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition
> >>>>> [0x00007fd39af45000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e6995e88> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
> >>>>> nid=0x3fb7 runnable [0x00007fd39b246000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e6886b58> (a
> >>>>> io.netty.channel.nio.SelectedSelectionKeySet)
> >>>>> - locked <0x00000000e689e610> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> >>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> >>>>> at
> >>>>>
> >>>>>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Yarn application state monitor" #73 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition
> >>>>> [0x00007fd39b8fb000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
> >>>>>
> >>>>> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
> >>>>> nid=0x3f77 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800
> >>>>> nid=0x3e6f
> >>>>> waiting on condition [0x00007fd39b547000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e677ae98> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd432896800 nid=0x3e6a waiting on condition
> >>>>> [0x00007fd39b9fc000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
> >>>>> at
> org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> >>>>> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000
> >>>>> nid=0x3e5e
> >>>>> runnable [0x00007fd3b80ff000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native
> Method)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "IPC Client (1489870911) connection to /10.209.23.74:9022 from
> hadoop"
> >>>>> #62
> >>>>> daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in
> >>>>> Object.wait()
> >>>>> [0x00007fd3b8501000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
> >>>>> - locked <0x00000000e66af0f8> (a
> >>>>> org.apache.hadoop.ipc.Client$Connection)
> >>>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
> >>>>>
> >>>>> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition
> >>>>> [0x00007fd3b8602000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e66e9230> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> >>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
> >>>>>
> >>>>> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000
> nid=0x3e2f
> >>>>> in
> >>>>> Object.wait() [0x00007fd3b8703000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
> >>>>> - locked <0x00000000e66e8190> (a java.util.TaskQueue)
> >>>>> at java.util.TimerThread.run(Timer.java:505)
> >>>>>
> >>>>> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
> >>>>> nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e67194f0> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54
> >>>>> daemon
> >>>>> prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
> >>>>> [0x00007fd3b8fb6000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> >>>>> at
> >>>>>
> >>>>>
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> >>>>> at
> >>>>>
> >>>>>
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> >>>>> - locked <0x00000000e671a620> (a java.lang.Object)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000000e6719020> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd432237000 nid=0x3e20 in Object.wait()
> [0x00007fd3b96d6000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
> >>>>> - locked <0x00000000e66609f0> (a java.util.TaskQueue)
> >>>>> at java.util.TimerThread.run(Timer.java:505)
> >>>>>
> >>>>> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
> >>>>> nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e6539dc8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
> >>>>> nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e6539dc8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
> >>>>> nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e6539dc8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
> >>>>> nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e6539dc8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
> >>>>> nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e6539dc8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
> >>>>> nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e6539dc8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
> >>>>> nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e6539dc8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon
> >>>>> prio=5
> >>>>> os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable
> >>>>> [0x00007fd3ba0de000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
> >>>>> at
> >>>>>
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> >>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> >>>>> at java.net.ServerSocket.accept(ServerSocket.java:513)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd431a45000 nid=0x3e0d in Object.wait()
> [0x00007fd3ba1df000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
> >>>>> - locked <0x00000000e6601270> (a java.util.TaskQueue)
> >>>>> at java.util.TimerThread.run(Timer.java:505)
> >>>>>
> >>>>> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd431a22800 nid=0x3e0c in Object.wait()
> [0x00007fd3ba2e0000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
> >>>>> - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
> >>>>> at java.util.TimerThread.run(Timer.java:505)
> >>>>>
> >>>>> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait()
> [0x00007fd3ba3e1000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
> >>>>> - locked <0x00000000e65380d8> (a java.util.TaskQueue)
> >>>>> at java.util.TimerThread.run(Timer.java:505)
> >>>>>
> >>>>> "New I/O server boss #6" #36 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd39c04b800
> >>>>> nid=0x3df8 runnable [0x00007fd3ba6e2000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000000e6543210> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #5" #35 daemon prio=5 os_prio=0
> tid=0x00007fd39c049800
> >>>>> nid=0x3df7 runnable [0x00007fd3ba7e3000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000000e6541ed8> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #4" #34 daemon prio=5 os_prio=0
> tid=0x00007fd39c04d800
> >>>>> nid=0x3df6 runnable [0x00007fd3ba8e4000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000000e6542568> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
> >>>>> nid=0x3df5 runnable [0x00007fd3ba9e5000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000000e657cbb8> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #2" #31 daemon prio=5 os_prio=0
> tid=0x00007fd39c029800
> >>>>> nid=0x3df4 runnable [0x00007fd3baae6000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000000e64badc0> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "New I/O worker #1" #30 daemon prio=5 os_prio=0
> tid=0x00007fd39c047000
> >>>>> nid=0x3df3 runnable [0x00007fd3babe7000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> >>>>> - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
> >>>>> - locked <0x00000000e64bb358> (a
> java.util.Collections$UnmodifiableSet)
> >>>>> - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
> >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> >>>>> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon
> prio=5
> >>>>> os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
> >>>>> [0x00007fd3baef9000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e64bbb40> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon
> prio=5
> >>>>> os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
> >>>>> [0x00007fd3baffa000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e64bbb40> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5
> >>>>> os_prio=0
> >>>>> tid=0x00007fd431f99000 nid=0x3de0 waiting on condition
> >>>>> [0x00007fd3bb0fb000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e64f49f8> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5
> >>>>> os_prio=0
> >>>>> tid=0x00007fd431f96800 nid=0x3ddf waiting on condition
> >>>>> [0x00007fd3bb1fc000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e64f49f8> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5
> >>>>> os_prio=0
> >>>>> tid=0x00007fd431f86000 nid=0x3dde waiting on condition
> >>>>> [0x00007fd3bb2fd000]
> >>>>>   java.lang.Thread.State: WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e64f49f8> (a
> >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> >>>>> at
> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> >>>>> at
> >>>>>
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >>>>> at
> >>>>>
> >>>>>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >>>>>
> >>>>> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
> >>>>> tid=0x00007fd431f0e800
> >>>>> nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>>> at java.lang.Thread.sleep(Native Method)
> >>>>> at
> akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> >>>>> at
> >>>>>
> >>>>>
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> >>>>> at
> >>>>>
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
> >>>>> nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e42808f8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
> >>>>> nid=0x3d75 waiting on condition [0x00007fd40c15d000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e42808f8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
> >>>>> nid=0x3d74 waiting on condition [0x00007fd40c25e000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e42808f8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
> >>>>> nid=0x3d73 waiting on condition [0x00007fd40c35f000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e42808f8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
> >>>>> nid=0x3d72 waiting on condition [0x00007fd40c460000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e42808f8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
> >>>>> nid=0x3d71 waiting on condition [0x00007fd40c561000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e42808f8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
> >>>>> nid=0x3d70 waiting on condition [0x00007fd40c662000]
> >>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>> - parking to wait for  <0x00000000e42808f8> (a
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>>> at
> >>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14
> daemon
> >>>>> prio=5
> >>>>> os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable
> >>>>> [0x00007fd40c966000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
> >>>>> at
> >>>>>
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> >>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> >>>>> at java.net.ServerSocket.accept(ServerSocket.java:513)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> >>>>> at
> >>>>>
> >>>>>
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> >>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>
> >>>>> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
> >>>>> nid=0x3d67 runnable [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0
> tid=0x00007fd4300c3800
> >>>>> nid=0x3d66 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0
> tid=0x00007fd4300c2000
> >>>>> nid=0x3d65 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0
> tid=0x00007fd4300bf000
> >>>>> nid=0x3d64 waiting on condition [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
> >>>>> nid=0x3d63 runnable [0x0000000000000000]
> >>>>>   java.lang.Thread.State: RUNNABLE
> >>>>>
> >>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800
> nid=0x3d62
> >>>>> in
> >>>>> Object.wait() [0x00007fd40f28e000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> >>>>> - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
> >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> >>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> >>>>>
> >>>>> "Reference Handler" #2 daemon prio=10 os_prio=0
> tid=0x00007fd430083800
> >>>>> nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> >>>>> - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
> >>>>>
> >>>>> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
> >>>>> Object.wait() [0x00007fd438220000]
> >>>>>   java.lang.Thread.State: WAITING (on object monitor)
> >>>>> at java.lang.Object.wait(Native Method)
> >>>>> at java.lang.Object.wait(Object.java:502)
> >>>>> at
> org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> >>>>> - locked <0x00000000fda1a8c0> (a
> org.apache.spark.scheduler.JobWaiter)
> >>>>> at
> >>>>>
> org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
> >>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
> >>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
> >>>>> at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
> >>>>> at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
> >>>>> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
> >>>>> at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
> >>>>> at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
> >>>>> at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
> >>>>> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
> >>>>> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
> >>>>> at
> >>>>>
> >>>>>
> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
> >>>>> at
> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
> >>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
> >>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
> >>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
> >>>>> at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
> >>>>> at $line24.$read$$iwC$$iwC.<init>(<console>:35)
> >>>>> at $line24.$read$$iwC.<init>(<console>:37)
> >>>>> at $line24.$read.<init>(<console>:39)
> >>>>> at $line24.$read$.<init>(<console>:43)
> >>>>> at $line24.$read$.<clinit>(<console>)
> >>>>> at $line24.$eval$.<init>(<console>:7)
> >>>>> at $line24.$eval$.<clinit>(<console>)
> >>>>> at $line24.$eval.$print(<console>)
> >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>> at
> >>>>>
> >>>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>>>> at
> >>>>>
> >>>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>> at java.lang.reflect.Method.invoke(Method.java:497)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
> >>>>> at
> >>>>>
> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> >>>>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
> >>>>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> >>>>> at
> >>>>>
> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
> >>>>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
> >>>>> at
> org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
> >>>>> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
> >>>>> at
> >>>>>
> >>>>> org.apache.spark.repl.SparkILoop.org
> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> >>>>> at
> >>>>>
> >>>>>
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> >>>>> at
> >>>>>
> >>>>> org.apache.spark.repl.SparkILoop.org
> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
> >>>>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
> >>>>> at org.apache.spark.repl.Main$.main(Main.scala:31)
> >>>>> at org.apache.spark.repl.Main.main(Main.scala)
> >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>> at
> >>>>>
> >>>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>>>> at
> >>>>>
> >>>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>> at java.lang.reflect.Method.invoke(Method.java:497)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
> >>>>> at
> >>>>>
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
> >>>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
> >>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
> >>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> >>>>>
> >>>>> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
> >>>>>
> >>>>> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000
> >>>>> nid=0x3d5c
> >>>>> runnable
> >>>>>
> >>>>> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800
> >>>>> nid=0x3d5d
> >>>>> runnable
> >>>>>
> >>>>> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800
> >>>>> nid=0x3d5e
> >>>>> runnable
> >>>>>
> >>>>> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000
> >>>>> nid=0x3d5f
> >>>>> runnable
> >>>>>
> >>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
> >>>>> waiting on condition
> >>>>>
> >>>>> JNI global references: 420
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust
> >>>>> <mi...@databricks.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Is it possible to jstack the executors and see where they are
> hanging?
> >>>>>>
> >>>>>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu,
> >>>>>>> 30GB),
> >>>>>>> executor memory 20GB, driver memory 10GB
> >>>>>>>
> >>>>>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data
> >>>>>>> spread
> >>>>>>> out over roughly 2,000 Parquet files and my queries frequently
> hang.
> >>>>>>> Simple
> >>>>>>> queries like "select count(*) from ..." on the entire data set work
> >>>>>>> ok.
> >>>>>>> Slightly more demanding ones with group by's and some aggregate
> >>>>>>> functions
> >>>>>>> (percentile_approx, avg, etc.) work ok as well, as long as I have
> some
> >>>>>>> criteria in my where clause to keep the number of rows down.
> >>>>>>>
> >>>>>>> Once I hit some limit on query complexity and rows processed, my
> >>>>>>> queries
> >>>>>>> start to hang.  I've left them for up to an hour without seeing any
> >>>>>>> progress.  No OOM's either - the job is just stuck.
> >>>>>>>
> >>>>>>> I've tried setting spark.sql.shuffle.partitions to 400 and even
> 800,
> >>>>>>> but
> >>>>>>> with the same results: usually near the end of the tasks (like 780
> of
> >>>>>>> 800
> >>>>>>> complete), progress just stops:
> >>>>>>>
> >>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task
> 788.0
> >>>>>>> in
> >>>>>>> stage 1.0 (TID 1618) in 800 ms on
> >>>>>>> ip-10-209-22-211.eu-west-1.compute.internal (748/800)
> >>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task
> 793.0
> >>>>>>> in
> >>>>>>> stage 1.0 (TID 1623) in 622 ms on
> >>>>>>> ip-10-105-12-41.eu-west-1.compute.internal
> >>>>>>> (749/800)
> >>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task
> 797.0
> >>>>>>> in
> >>>>>>> stage 1.0 (TID 1627) in 616 ms on
> >>>>>>> ip-10-90-2-201.eu-west-1.compute.internal
> >>>>>>> (750/800)
> >>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task
> 799.0
> >>>>>>> in
> >>>>>>> stage 1.0 (TID 1629) in 611 ms on
> >>>>>>> ip-10-90-2-201.eu-west-1.compute.internal
> >>>>>>> (751/800)
> >>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task
> 795.0
> >>>>>>> in
> >>>>>>> stage 1.0 (TID 1625) in 669 ms on
> >>>>>>> ip-10-105-12-41.eu-west-1.compute.internal
> >>>>>>> (752/800)
> >>>>>>>
> >>>>>>> ^^^^^^^ this is where it stays forever
> >>>>>>>
> >>>>>>> Looking at the Spark UI, several of the executors still list active
> >>>>>>> tasks.  I do see that the Shuffle Read for executors that don't
> have
> >>>>>>> any
> >>>>>>> tasks remaining is around 100MB, whereas it's more like 10MB for
> the
> >>>>>>> executors that still have tasks.
> >>>>>>>
> >>>>>>> The first stage, mapPartitions, always completes fine.  It's the
> >>>>>>> second
> >>>>>>> stage (takeOrdered), that hangs.
> >>>>>>>
> >>>>>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
> >>>>>>> encountered it when using JSON files (instead of Parquet).
> >>>>>>>
> >>>>>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries
> I do
> >>>>>>> are having this issue.
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Christian Perez
> >>>> Silicon Valley Data Science
> >>>> Data Analyst
> >>>> christian@svds.com
> >>>> @cp_phd
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >>>> For additional commands, e-mail: user-help@spark.apache.org
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Christian Perez
> >> Silicon Valley Data Science
> >> Data Analyst
> >> christian@svds.com
> >> @cp_phd
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: user-help@spark.apache.org
> >>
> >
>
>
>
> --
> Christian Perez
> Silicon Valley Data Science
> Data Analyst
> christian@svds.com
> @cp_phd
>

Re: Spark SQL queries hang forever

Posted by Christian Perez <ch...@svds.com>.
I found that rewriting our tables in Impala (originally Hive) solved
the problem. The files are bigger so that there are 180 instead of
718.

I also think that turning the shuffle.partitions down to 100 on the
original set prevented Spark from hanging, but haven't tried to
reproduce since finding this workaround.

Perhaps there's an issue with either a) hidden Spark issues with the
particular parquet files or b) subtle bugs that appear later when
Spark processes many files. **A shuffle hang occurred with >800 text
files of same data.**

On Wed, Apr 1, 2015 at 6:39 AM, Marius Soutier <mp...@gmail.com> wrote:
> I have similiar problem since upgrading to 1.2.1.
>
> SparkSQL queries on small, simple Parquet files take up to 10 minutes. After an initial hang, it seems to work fine for a while.
>
> The problematic stage is RangePartitioner at Exchange.scala:88.
>
>
>> On 31.03.2015, at 21:55, Christian Perez <ch...@svds.com> wrote:
>>
>> I think I've seen a timeout in one of the executor logs... but I'm
>> usually not that patient.
>>
>> If I kill the executor running the last task started on spark UI, 2
>> out of 3 times it will retry and complete successfully.
>>
>> I've found that everything works when running in local mode versus
>> YARN. Have not tried Spark standalone. Something (block fetcher?) is
>> hanging during shuffle when distributed...
>>
>> On Tue, Mar 31, 2015 at 12:10 AM, Zsolt Tóth <to...@gmail.com> wrote:
>>> I have a very similar problem with collect() but only on relatively large
>>> input data, on Amazon EMR. Look for my thread for details if interested.
>>> I get "connection timed out" exception after a 15-20 minutes in one of the
>>> container logs - does it happen in your case?
>>>
>>> 2015-03-30 23:34 GMT+02:00 Christian Perez <ch...@svds.com>:
>>>>
>>>> I'm having the exact same problem (hanging) with takeOrdered (using
>>>> PySpark's orderBy on a DataFrame.)
>>>>
>>>> On Thu, Mar 26, 2015 at 7:21 PM, Jon Chase <jo...@gmail.com> wrote:
>>>>> Here are the jstacks from a 3 worker cluster.  Two of the workers are
>>>>> sitting with 4 active tasks, the other has no active tasks.  Not sure if
>>>>> it
>>>>> makes a difference, but I only see an ExecutorLauncher instance via JPS
>>>>> on
>>>>> the worker that has no active tasks.
>>>>>
>>>>> Here's the first stack dump, this machine has 4 active tasks:
>>>>>
>>>>> 2015-03-27 02:11:46
>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>>> mode):
>>>>>
>>>>> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
>>>>> nid=0x43a3 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
>>>>> [0x00007f91bad9a000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589884fc0> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
>>>>> nid=0x42e2 runnable [0x00007f91b4c54000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005898aa000> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
>>>>> nid=0x42e1 runnable [0x00007f91b5d56000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058ad5b5f8> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
>>>>> nid=0x42e0 runnable [0x00007f91b5e57000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005898aa0c0> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
>>>>> nid=0x42df runnable [0x00007f91b5f58000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058ad5b6b8> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91d0773800 nid=0x427b waiting on condition
>>>>> [0x00007f91b6258000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006cfda6800> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition
>>>>> [0x00007f91b6b5a000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
>>>>> waiting on condition [0x00007f91b6c5b000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058af8e0b8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
>>>>> runnable [0x00007f91baf9c000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0
>>>>> tid=0x00007f91d8118800
>>>>> nid=0x41ee waiting on condition [0x00007f91bae9b000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
>>>>> nid=0x41ed runnable [0x00007f91baa97000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589224af0> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91d0019000 nid=0x41ea waiting on condition
>>>>> [0x00007f91bac98000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006c8837750> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91d0016800 nid=0x41e9 waiting on condition
>>>>> [0x00007f91ba16d000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006cc53bff0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91d0015000 nid=0x41e8 waiting on condition
>>>>> [0x00007f91b9f6b000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006cfda69b0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
>>>>> nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>>>>
>>>>> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
>>>>> nid=0x41c6 runnable [0x00007f91bbac7000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058984ee60> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition
>>>>> [0x00007f91bbbc8000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005898f7290> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x000000058974cb00> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x0000000589904038> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x0000000589904038> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91ed92d800
>>>>> nid=0x41bf waiting on condition [0x00007f91bab98000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589884e80> (a
>>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91ed92c800
>>>>> nid=0x41be waiting on condition [0x00007f91ba370000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O server boss #12" #39 daemon prio=5 os_prio=0
>>>>> tid=0x0000000000a02000
>>>>> nid=0x41bd runnable [0x00007f91b9564000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
>>>>> nid=0x41bc runnable [0x00007f91b9665000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
>>>>> nid=0x41bb runnable [0x00007f91b9766000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
>>>>> nid=0x41ba runnable [0x00007f91b9867000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
>>>>> nid=0x41b9 runnable [0x00007f91b9968000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
>>>>> nid=0x41b8 runnable [0x00007f91b9c69000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
>>>>> [0x00007f91b9d6a000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589884f20> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
>>>>> [0x00007f91b9e6b000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589884f20> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>>> at
>>>>>
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
>>>>> nid=0x419e runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
>>>>> nid=0x419d waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
>>>>> nid=0x419c waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
>>>>> nid=0x419b waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
>>>>> nid=0x419a runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>>>>> tid=0x00007f91ec101000 nid=0x4199 waiting on condition
>>>>> [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198
>>>>> in
>>>>> Object.wait() [0x00007f91d4254000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>>
>>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
>>>>> nid=0x4197 in Object.wait() [0x00007f91d4355000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>> - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>>>>>
>>>>> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
>>>>> condition [0x00007f91f1fdf000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589a1b5f0> (a
>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at
>>>>>
>>>>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>>>> at scala.concurrent.Await$.ready(package.scala:86)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>>>>
>>>>> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>>>>>
>>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
>>>>> nid=0x4191 runnable
>>>>>
>>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
>>>>> nid=0x4192 runnable
>>>>>
>>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
>>>>> nid=0x4193 runnable
>>>>>
>>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
>>>>> nid=0x4194 runnable
>>>>>
>>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
>>>>> nid=0x4195 runnable
>>>>>
>>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
>>>>> waiting on condition
>>>>>
>>>>> JNI global references: 319
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Here's the next worker, also for active tasks:
>>>>>
>>>>> 2015-03-27 02:12:54
>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>>> mode):
>>>>>
>>>>> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
>>>>> nid=0x44f1 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
>>>>> [0x00007fcbbf87f000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589510428> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbe4028800 nid=0x445f waiting on condition
>>>>> [0x00007fcbbf97f000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006126cd358> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
>>>>> nid=0x4459 runnable [0x00007fcbc88c7000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a6b8c98> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
>>>>> nid=0x4458 runnable [0x00007fcbc7897000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a6e5a18> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
>>>>> nid=0x4457 runnable [0x00007fcbc7998000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058aade110> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
>>>>> nid=0x4456 runnable [0x00007fcbc7b9a000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058aade1d0> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition
>>>>> [0x00007fcbc4886000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000615539dd8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbe4165000 nid=0x4453 waiting on condition
>>>>> [0x00007fcbca620000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000617bd4238> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbe4164000 nid=0x4452 waiting on condition
>>>>> [0x00007fcbc87c5000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006126cd538> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbdcff1800 nid=0x436e waiting on condition
>>>>> [0x00007fcbc4485000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
>>>>> waiting on condition [0x00007fcbc4586000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058adabe70> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
>>>>> runnable [0x00007fcbc7e9d000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
>>>>> tid=0x00007fcbf9608800
>>>>> nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
>>>>> nid=0x4364 runnable [0x00007fcbc8ac9000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589116ae0> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
>>>>> nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>>>>
>>>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
>>>>> nid=0x4340 runnable [0x00007fcbc94f3000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058964c068> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition
>>>>> [0x00007fcbc97f6000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058a6fa058> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x000000058928c438> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x000000058928c438> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x000000058928c5a0> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbdc29d000
>>>>> nid=0x433a waiting on condition [0x00007fcbc85c4000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbe4474800
>>>>> nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058a70a468> (a
>>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbdc29a000
>>>>> nid=0x4337 runnable [0x00007fcbc6e8f000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
>>>>> nid=0x4336 runnable [0x00007fcbc6f90000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
>>>>> nid=0x4335 runnable [0x00007fcbc7091000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
>>>>> nid=0x4334 runnable [0x00007fcbc7392000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
>>>>> nid=0x4333 runnable [0x00007fcbc7493000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
>>>>> nid=0x4332 runnable [0x00007fcbc7594000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
>>>>> [0x00007fcbc7695000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005893c1820> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
>>>>> [0x00007fcbc7796000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005893c1820> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbf995d800 nid=0x432a waiting on condition
>>>>> [0x00007fcbc7d9c000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>>> at
>>>>>
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
>>>>> nid=0x4318 runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
>>>>> nid=0x4317 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
>>>>> nid=0x4316 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
>>>>> nid=0x4315 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
>>>>> nid=0x4314 runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>>>>> tid=0x00007fcbf8103800 nid=0x4313 waiting on condition
>>>>> [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312
>>>>> in
>>>>> Object.wait() [0x00007fcbe1be7000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>>
>>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
>>>>> nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>> - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>>>>>
>>>>> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
>>>>> condition [0x00007fcbff9b9000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589585398> (a
>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at
>>>>>
>>>>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>>>> at scala.concurrent.Await$.ready(package.scala:86)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>>>>
>>>>> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>>>>>
>>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
>>>>> nid=0x430b runnable
>>>>>
>>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
>>>>> nid=0x430c runnable
>>>>>
>>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
>>>>> nid=0x430d runnable
>>>>>
>>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
>>>>> nid=0x430e runnable
>>>>>
>>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
>>>>> nid=0x430f runnable
>>>>>
>>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
>>>>> waiting on condition
>>>>>
>>>>> JNI global references: 375
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> And the final worker, no active tasks:
>>>>>
>>>>>
>>>>> 2015-03-27 02:13:26
>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>>> mode):
>>>>>
>>>>> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
>>>>> nid=0x3c5b waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
>>>>> [0x00007f36d2e33000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005895676e8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
>>>>> nid=0x3b88 runnable [0x00007f36dafb9000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a86a7d8> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
>>>>> nid=0x3b87 runnable [0x00007f36d7138000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058abea6c8> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
>>>>> nid=0x3b86 runnable [0x00007f36daeb8000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a86a898> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
>>>>> nid=0x3b85 runnable [0x00007f36d7e3b000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058abea788> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
>>>>> tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition
>>>>> [0x00007f36d7a39000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
>>>>> waiting on condition [0x00007f36d7b3a000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058aecd008> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
>>>>> runnable [0x00007f36db4be000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
>>>>> tid=0x00007f36f8025000
>>>>> nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
>>>>> nid=0x3a96 runnable [0x00007f36dc0ea000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589202c58> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
>>>>> nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>>>>
>>>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
>>>>> nid=0x3a6c runnable [0x00007f36dcb14000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a825c18> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
>>>>> tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition
>>>>> [0x00007f36dce17000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005897334d0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
>>>>> tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000005896b0d20> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
>>>>> tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
>>>>> tid=0x00007f36f03a1000
>>>>> nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
>>>>> tid=0x00007f370d9b2000
>>>>> nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005895675a8> (a
>>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
>>>>> tid=0x00007f36f8713000
>>>>> nid=0x3a63 runnable [0x00007f36da4b0000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
>>>>> nid=0x3a62 runnable [0x00007f36da5b1000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
>>>>> nid=0x3a61 runnable [0x00007f36da6b2000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
>>>>> nid=0x3a60 runnable [0x00007f36da7b3000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
>>>>> nid=0x3a5f runnable [0x00007f36da8b4000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
>>>>> nid=0x3a5e runnable [0x00007f36da9b5000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
>>>>> [0x00007f36dacb6000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589567648> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
>>>>> [0x00007f36dadb7000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589567648> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>>>>> tid=0x00007f370d97c000 nid=0x3a56 waiting on condition
>>>>> [0x00007f36db3bd000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>>> at
>>>>>
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
>>>>> nid=0x3a43 runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
>>>>> nid=0x3a42 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
>>>>> nid=0x3a41 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
>>>>> nid=0x3a40 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
>>>>> nid=0x3a3f runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>>>>> tid=0x00007f370c101800 nid=0x3a3e waiting on condition
>>>>> [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d
>>>>> in
>>>>> Object.wait() [0x00007f36f51e5000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>>
>>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
>>>>> nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>> - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>>>>>
>>>>> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
>>>>> condition [0x00007f3712fda000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058a974ab8> (a
>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at
>>>>>
>>>>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>>>> at scala.concurrent.Await$.ready(package.scala:86)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>>>>
>>>>> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>>>>>
>>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
>>>>> nid=0x3a34 runnable
>>>>>
>>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
>>>>> nid=0x3a35 runnable
>>>>>
>>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
>>>>> nid=0x3a36 runnable
>>>>>
>>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
>>>>> nid=0x3a37 runnable
>>>>>
>>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
>>>>> nid=0x3a3a runnable
>>>>>
>>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
>>>>> waiting on condition
>>>>>
>>>>> JNI global references: 486
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> And the driver:
>>>>>
>>>>> 2015-03-27 02:20:47
>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>>> mode):
>>>>>
>>>>> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
>>>>> nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
>>>>> nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
>>>>> nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
>>>>> nid=0x41d0 waiting on condition [0x00007fd38feef000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition
>>>>> [0x00007fd38fff0000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000fd545eb8> (a
>>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition
>>>>> [0x00007fd3b8400000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
>>>>> nid=0x417f waiting on condition [0x00007fd398935000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "task-result-getter-3" #107 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3b003f800
>>>>> nid=0x409c waiting on condition [0x00007fd3902f1000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "task-result-getter-2" #106 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3b003d000
>>>>> nid=0x409b waiting on condition [0x00007fd3903f2000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "task-result-getter-1" #105 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3b003b800
>>>>> nid=0x409a waiting on condition [0x00007fd3904f3000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "task-result-getter-0" #104 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3b0030800
>>>>> nid=0x4098 waiting on condition [0x00007fd3905f4000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
>>>>> tid=0x00007fd3cc00c800
>>>>> nid=0x4093 waiting on condition [0x00007fd3936f8000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
>>>>> nid=0x4092 runnable [0x00007fd398b37000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e685f320> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
>>>>> nid=0x4091 runnable [0x00007fd398a36000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6871c30> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
>>>>> nid=0x4090 runnable [0x00007fd39903a000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6873520> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3e0011800 nid=0x4056 waiting on condition
>>>>> [0x00007fd398733000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3dc001800 nid=0x4055 waiting on condition
>>>>> [0x00007fd398834000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e7f19070> (a
>>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fd3a400d800 nid=0x4051 waiting on condition
>>>>> [0x00007fd398e38000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd432c05000 nid=0x404f waiting on condition
>>>>> [0x00007fd398f39000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66fe220> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>>
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd43297c800 nid=0x4042 waiting on condition
>>>>> [0x00007fd39973b000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e740fa58> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>>>> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4329ac000 nid=0x4041 waiting on condition
>>>>> [0x00007fd39983c000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e740fce8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4328e6800 nid=0x403b waiting on condition
>>>>> [0x00007fd399c3e000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e737a0c8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>>>> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4328eb800 nid=0x403a waiting on condition
>>>>> [0x00007fd399d3f000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e737add8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at
>>>>>
>>>>> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>>>>>
>>>>> "ResponseProcessor for block
>>>>> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
>>>>> prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
>>>>> [0x00007fd39a941000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>>>> at
>>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>>>>> at
>>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>>>>> at
>>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>>>>>
>>>>> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
>>>>> nid=0x3fda waiting on condition [0x00007fd39aa42000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e58145f0> (a
>>>>> java.util.concurrent.Semaphore$NonfairSync)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>>>> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>>>>>
>>>>> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd43258c000
>>>>> nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at
>>>>>
>>>>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
>>>>> at
>>>>>
>>>>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>>>> at
>>>>>
>>>>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>>>> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>>>> at
>>>>>
>>>>> org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
>>>>> at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>>>>>
>>>>> "DataStreamer for file
>>>>> /spark-logs/application_1427416187821_0004.inprogress
>>>>> block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84
>>>>> daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in
>>>>> Object.wait()
>>>>> [0x00007fd39ae44000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
>>>>> - locked <0x00000000e6994f30> (a java.util.LinkedList)
>>>>>
>>>>> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition
>>>>> [0x00007fd39af45000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6995e88> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
>>>>> nid=0x3fb7 runnable [0x00007fd39b246000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6886b58> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Yarn application state monitor" #73 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition
>>>>> [0x00007fd39b8fb000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>>>>>
>>>>> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
>>>>> nid=0x3f77 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800
>>>>> nid=0x3e6f
>>>>> waiting on condition [0x00007fd39b547000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e677ae98> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd432896800 nid=0x3e6a waiting on condition
>>>>> [0x00007fd39b9fc000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
>>>>> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>>>>> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000
>>>>> nid=0x3e5e
>>>>> runnable [0x00007fd3b80ff000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop"
>>>>> #62
>>>>> daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in
>>>>> Object.wait()
>>>>> [0x00007fd3b8501000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
>>>>> - locked <0x00000000e66af0f8> (a
>>>>> org.apache.hadoop.ipc.Client$Connection)
>>>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>>>>>
>>>>> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition
>>>>> [0x00007fd3b8602000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66e9230> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
>>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>>>>>
>>>>> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f
>>>>> in
>>>>> Object.wait() [0x00007fd3b8703000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000000e66e8190> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
>>>>> nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54
>>>>> daemon
>>>>> prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
>>>>> [0x00007fd3b8fb6000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>>
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>>>>> at
>>>>>
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>>>>> - locked <0x00000000e671a620> (a java.lang.Object)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000000e66609f0> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
>>>>> nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
>>>>> nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
>>>>> nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
>>>>> nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
>>>>> nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
>>>>> nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
>>>>> nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable
>>>>> [0x00007fd3ba0de000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at
>>>>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:513)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000000e6601270> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000000e65380d8> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "New I/O server boss #6" #36 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd39c04b800
>>>>> nid=0x3df8 runnable [0x00007fd3ba6e2000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
>>>>> nid=0x3df7 runnable [0x00007fd3ba7e3000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
>>>>> nid=0x3df6 runnable [0x00007fd3ba8e4000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
>>>>> nid=0x3df5 runnable [0x00007fd3ba9e5000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
>>>>> nid=0x3df4 runnable [0x00007fd3baae6000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
>>>>> nid=0x3df3 runnable [0x00007fd3babe7000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
>>>>> os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
>>>>> [0x00007fd3baef9000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64bbb40> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
>>>>> os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
>>>>> [0x00007fd3baffa000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64bbb40> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fd431f99000 nid=0x3de0 waiting on condition
>>>>> [0x00007fd3bb0fb000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fd431f96800 nid=0x3ddf waiting on condition
>>>>> [0x00007fd3bb1fc000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fd431f86000 nid=0x3dde waiting on condition
>>>>> [0x00007fd3bb2fd000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd431f0e800
>>>>> nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>>> at
>>>>>
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
>>>>> nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
>>>>> nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
>>>>> nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
>>>>> nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
>>>>> nid=0x3d72 waiting on condition [0x00007fd40c460000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
>>>>> nid=0x3d71 waiting on condition [0x00007fd40c561000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
>>>>> nid=0x3d70 waiting on condition [0x00007fd40c662000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable
>>>>> [0x00007fd40c966000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at
>>>>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:513)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
>>>>> nid=0x3d67 runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
>>>>> nid=0x3d66 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
>>>>> nid=0x3d65 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
>>>>> nid=0x3d64 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
>>>>> nid=0x3d63 runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62
>>>>> in
>>>>> Object.wait() [0x00007fd40f28e000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>>
>>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
>>>>> nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>> - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>>>>>
>>>>> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
>>>>> Object.wait() [0x00007fd438220000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
>>>>> - locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
>>>>> at
>>>>> org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
>>>>> at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
>>>>> at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
>>>>> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
>>>>> at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
>>>>> at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
>>>>> at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
>>>>> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
>>>>> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
>>>>> at
>>>>>
>>>>> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
>>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
>>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>>>>> at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
>>>>> at $line24.$read$$iwC$$iwC.<init>(<console>:35)
>>>>> at $line24.$read$$iwC.<init>(<console>:37)
>>>>> at $line24.$read.<init>(<console>:39)
>>>>> at $line24.$read$.<init>(<console>:43)
>>>>> at $line24.$read$.<clinit>(<console>)
>>>>> at $line24.$eval$.<init>(<console>:7)
>>>>> at $line24.$eval$.<clinit>(<console>)
>>>>> at $line24.$eval.$print(<console>)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> at
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>>>>> at
>>>>> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>>>>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>>>>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>>>>> at
>>>>> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
>>>>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
>>>>> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
>>>>> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>>>> at
>>>>>
>>>>> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>>>>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
>>>>> at org.apache.spark.repl.Main$.main(Main.scala:31)
>>>>> at org.apache.spark.repl.Main.main(Main.scala)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> at
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>>>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>
>>>>> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>>>>>
>>>>> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000
>>>>> nid=0x3d5c
>>>>> runnable
>>>>>
>>>>> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800
>>>>> nid=0x3d5d
>>>>> runnable
>>>>>
>>>>> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800
>>>>> nid=0x3d5e
>>>>> runnable
>>>>>
>>>>> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000
>>>>> nid=0x3d5f
>>>>> runnable
>>>>>
>>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
>>>>> waiting on condition
>>>>>
>>>>> JNI global references: 420
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust
>>>>> <mi...@databricks.com>
>>>>> wrote:
>>>>>>
>>>>>> Is it possible to jstack the executors and see where they are hanging?
>>>>>>
>>>>>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:
>>>>>>>
>>>>>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu,
>>>>>>> 30GB),
>>>>>>> executor memory 20GB, driver memory 10GB
>>>>>>>
>>>>>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data
>>>>>>> spread
>>>>>>> out over roughly 2,000 Parquet files and my queries frequently hang.
>>>>>>> Simple
>>>>>>> queries like "select count(*) from ..." on the entire data set work
>>>>>>> ok.
>>>>>>> Slightly more demanding ones with group by's and some aggregate
>>>>>>> functions
>>>>>>> (percentile_approx, avg, etc.) work ok as well, as long as I have some
>>>>>>> criteria in my where clause to keep the number of rows down.
>>>>>>>
>>>>>>> Once I hit some limit on query complexity and rows processed, my
>>>>>>> queries
>>>>>>> start to hang.  I've left them for up to an hour without seeing any
>>>>>>> progress.  No OOM's either - the job is just stuck.
>>>>>>>
>>>>>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800,
>>>>>>> but
>>>>>>> with the same results: usually near the end of the tasks (like 780 of
>>>>>>> 800
>>>>>>> complete), progress just stops:
>>>>>>>
>>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0
>>>>>>> in
>>>>>>> stage 1.0 (TID 1618) in 800 ms on
>>>>>>> ip-10-209-22-211.eu-west-1.compute.internal (748/800)
>>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0
>>>>>>> in
>>>>>>> stage 1.0 (TID 1623) in 622 ms on
>>>>>>> ip-10-105-12-41.eu-west-1.compute.internal
>>>>>>> (749/800)
>>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0
>>>>>>> in
>>>>>>> stage 1.0 (TID 1627) in 616 ms on
>>>>>>> ip-10-90-2-201.eu-west-1.compute.internal
>>>>>>> (750/800)
>>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0
>>>>>>> in
>>>>>>> stage 1.0 (TID 1629) in 611 ms on
>>>>>>> ip-10-90-2-201.eu-west-1.compute.internal
>>>>>>> (751/800)
>>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0
>>>>>>> in
>>>>>>> stage 1.0 (TID 1625) in 669 ms on
>>>>>>> ip-10-105-12-41.eu-west-1.compute.internal
>>>>>>> (752/800)
>>>>>>>
>>>>>>> ^^^^^^^ this is where it stays forever
>>>>>>>
>>>>>>> Looking at the Spark UI, several of the executors still list active
>>>>>>> tasks.  I do see that the Shuffle Read for executors that don't have
>>>>>>> any
>>>>>>> tasks remaining is around 100MB, whereas it's more like 10MB for the
>>>>>>> executors that still have tasks.
>>>>>>>
>>>>>>> The first stage, mapPartitions, always completes fine.  It's the
>>>>>>> second
>>>>>>> stage (takeOrdered), that hangs.
>>>>>>>
>>>>>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
>>>>>>> encountered it when using JSON files (instead of Parquet).
>>>>>>>
>>>>>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
>>>>>>> are having this issue.
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Christian Perez
>>>> Silicon Valley Data Science
>>>> Data Analyst
>>>> christian@svds.com
>>>> @cp_phd
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>
>>
>>
>>
>> --
>> Christian Perez
>> Silicon Valley Data Science
>> Data Analyst
>> christian@svds.com
>> @cp_phd
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>



-- 
Christian Perez
Silicon Valley Data Science
Data Analyst
christian@svds.com
@cp_phd

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark SQL queries hang forever

Posted by Marius Soutier <mp...@gmail.com>.
I have similiar problem since upgrading to 1.2.1.

SparkSQL queries on small, simple Parquet files take up to 10 minutes. After an initial hang, it seems to work fine for a while.

The problematic stage is RangePartitioner at Exchange.scala:88.


> On 31.03.2015, at 21:55, Christian Perez <ch...@svds.com> wrote:
> 
> I think I've seen a timeout in one of the executor logs... but I'm
> usually not that patient.
> 
> If I kill the executor running the last task started on spark UI, 2
> out of 3 times it will retry and complete successfully.
> 
> I've found that everything works when running in local mode versus
> YARN. Have not tried Spark standalone. Something (block fetcher?) is
> hanging during shuffle when distributed...
> 
> On Tue, Mar 31, 2015 at 12:10 AM, Zsolt Tóth <to...@gmail.com> wrote:
>> I have a very similar problem with collect() but only on relatively large
>> input data, on Amazon EMR. Look for my thread for details if interested.
>> I get "connection timed out" exception after a 15-20 minutes in one of the
>> container logs - does it happen in your case?
>> 
>> 2015-03-30 23:34 GMT+02:00 Christian Perez <ch...@svds.com>:
>>> 
>>> I'm having the exact same problem (hanging) with takeOrdered (using
>>> PySpark's orderBy on a DataFrame.)
>>> 
>>> On Thu, Mar 26, 2015 at 7:21 PM, Jon Chase <jo...@gmail.com> wrote:
>>>> Here are the jstacks from a 3 worker cluster.  Two of the workers are
>>>> sitting with 4 active tasks, the other has no active tasks.  Not sure if
>>>> it
>>>> makes a difference, but I only see an ExecutorLauncher instance via JPS
>>>> on
>>>> the worker that has no active tasks.
>>>> 
>>>> Here's the first stack dump, this machine has 4 active tasks:
>>>> 
>>>> 2015-03-27 02:11:46
>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>> mode):
>>>> 
>>>> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
>>>> nid=0x43a3 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
>>>> os_prio=0
>>>> tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
>>>> [0x00007f91bad9a000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000589884fc0> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
>>>> nid=0x42e2 runnable [0x00007f91b4c54000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000005898aa000> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
>>>> nid=0x42e1 runnable [0x00007f91b5d56000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058ad5b5f8> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
>>>> nid=0x42e0 runnable [0x00007f91b5e57000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000005898aa0c0> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
>>>> nid=0x42df runnable [0x00007f91b5f58000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058ad5b6b8> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
>>>> tid=0x00007f91d0773800 nid=0x427b waiting on condition
>>>> [0x00007f91b6258000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000006cfda6800> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>> at
>>>> 
>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>> at
>>>> 
>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
>>>> tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition
>>>> [0x00007f91b6b5a000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
>>>> waiting on condition [0x00007f91b6c5b000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x000000058af8e0b8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
>>>> runnable [0x00007f91baf9c000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>> at
>>>> 
>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>> at
>>>> 
>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0
>>>> tid=0x00007f91d8118800
>>>> nid=0x41ee waiting on condition [0x00007f91bae9b000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
>>>> nid=0x41ed runnable [0x00007f91baa97000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x0000000589224af0> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
>>>> tid=0x00007f91d0019000 nid=0x41ea waiting on condition
>>>> [0x00007f91bac98000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000006c8837750> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>> at
>>>> 
>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>> at
>>>> 
>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
>>>> tid=0x00007f91d0016800 nid=0x41e9 waiting on condition
>>>> [0x00007f91ba16d000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000006cc53bff0> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>> at
>>>> 
>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>> at
>>>> 
>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
>>>> tid=0x00007f91d0015000 nid=0x41e8 waiting on condition
>>>> [0x00007f91b9f6b000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000006cfda69b0> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>> at
>>>> 
>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>> at
>>>> 
>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
>>>> nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>>> 
>>>> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
>>>> nid=0x41c6 runnable [0x00007f91bbac7000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058984ee60> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
>>>> tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition
>>>> [0x00007f91bbbc8000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005898f7290> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
>>>> tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>> - locked <0x000000058974cb00> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> 
>>>> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
>>>> tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0x0000000589904038> (a java.util.TaskQueue)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>> - locked <0x0000000589904038> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> 
>>>> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
>>>> tid=0x00007f91ed92d800
>>>> nid=0x41bf waiting on condition [0x00007f91bab98000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000589884e80> (a
>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0
>>>> tid=0x00007f91ed92c800
>>>> nid=0x41be waiting on condition [0x00007f91ba370000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>> 
>>>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>>> at
>>>> 
>>>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O server boss #12" #39 daemon prio=5 os_prio=0
>>>> tid=0x0000000000a02000
>>>> nid=0x41bd runnable [0x00007f91b9564000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
>>>> nid=0x41bc runnable [0x00007f91b9665000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
>>>> nid=0x41bb runnable [0x00007f91b9766000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
>>>> nid=0x41ba runnable [0x00007f91b9867000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
>>>> nid=0x41b9 runnable [0x00007f91b9968000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
>>>> nid=0x41b8 runnable [0x00007f91b9c69000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon
>>>> prio=5
>>>> os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
>>>> [0x00007f91b9d6a000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000589884f20> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon
>>>> prio=5
>>>> os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
>>>> [0x00007f91b9e6b000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000589884f20> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>>>> tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>> at
>>>> 
>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>> at
>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
>>>> nid=0x419e runnable [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
>>>> nid=0x419d waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
>>>> nid=0x419c waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
>>>> nid=0x419b waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
>>>> nid=0x419a runnable [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>>>> tid=0x00007f91ec101000 nid=0x4199 waiting on condition
>>>> [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198
>>>> in
>>>> Object.wait() [0x00007f91d4254000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>> - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>> 
>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
>>>> nid=0x4197 in Object.wait() [0x00007f91d4355000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>> - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>>>> 
>>>> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
>>>> condition [0x00007f91f1fdf000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000589a1b5f0> (a
>>>> java.util.concurrent.CountDownLatch$Sync)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>> at
>>>> 
>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>>> at
>>>> 
>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>> at
>>>> 
>>>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>>> at scala.concurrent.Await$.ready(package.scala:86)
>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>>> at
>>>> 
>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>>> at
>>>> 
>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>> at
>>>> 
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>> at
>>>> 
>>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>>> 
>>>> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>>>> 
>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
>>>> nid=0x4191 runnable
>>>> 
>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
>>>> nid=0x4192 runnable
>>>> 
>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
>>>> nid=0x4193 runnable
>>>> 
>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
>>>> nid=0x4194 runnable
>>>> 
>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
>>>> nid=0x4195 runnable
>>>> 
>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
>>>> waiting on condition
>>>> 
>>>> JNI global references: 319
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Here's the next worker, also for active tasks:
>>>> 
>>>> 2015-03-27 02:12:54
>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>> mode):
>>>> 
>>>> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
>>>> nid=0x44f1 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
>>>> os_prio=0
>>>> tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
>>>> [0x00007fcbbf87f000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000589510428> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbe4028800 nid=0x445f waiting on condition
>>>> [0x00007fcbbf97f000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000006126cd358> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>> at
>>>> 
>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>> at
>>>> 
>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
>>>> nid=0x4459 runnable [0x00007fcbc88c7000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058a6b8c98> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
>>>> nid=0x4458 runnable [0x00007fcbc7897000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058a6e5a18> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
>>>> nid=0x4457 runnable [0x00007fcbc7998000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058aade110> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
>>>> nid=0x4456 runnable [0x00007fcbc7b9a000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058aade1d0> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition
>>>> [0x00007fcbc4886000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000615539dd8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>> at
>>>> 
>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>> at
>>>> 
>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbe4165000 nid=0x4453 waiting on condition
>>>> [0x00007fcbca620000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000617bd4238> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>> at
>>>> 
>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>> at
>>>> 
>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbe4164000 nid=0x4452 waiting on condition
>>>> [0x00007fcbc87c5000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000006126cd538> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>> at
>>>> 
>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>> at
>>>> 
>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>> at
>>>> 
>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbdcff1800 nid=0x436e waiting on condition
>>>> [0x00007fcbc4485000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
>>>> waiting on condition [0x00007fcbc4586000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x000000058adabe70> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
>>>> runnable [0x00007fcbc7e9d000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>> at
>>>> 
>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>> at
>>>> 
>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
>>>> tid=0x00007fcbf9608800
>>>> nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
>>>> nid=0x4364 runnable [0x00007fcbc8ac9000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x0000000589116ae0> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
>>>> nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>>> 
>>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
>>>> nid=0x4340 runnable [0x00007fcbc94f3000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058964c068> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition
>>>> [0x00007fcbc97f6000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x000000058a6fa058> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0x000000058928c438> (a java.util.TaskQueue)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>> - locked <0x000000058928c438> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> 
>>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>> - locked <0x000000058928c5a0> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> 
>>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbdc29d000
>>>> nid=0x433a waiting on condition [0x00007fcbc85c4000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>> 
>>>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>>> at
>>>> 
>>>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbe4474800
>>>> nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x000000058a70a468> (a
>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbdc29a000
>>>> nid=0x4337 runnable [0x00007fcbc6e8f000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
>>>> nid=0x4336 runnable [0x00007fcbc6f90000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
>>>> nid=0x4335 runnable [0x00007fcbc7091000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
>>>> nid=0x4334 runnable [0x00007fcbc7392000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
>>>> nid=0x4333 runnable [0x00007fcbc7493000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
>>>> nid=0x4332 runnable [0x00007fcbc7594000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon
>>>> prio=5
>>>> os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
>>>> [0x00007fcbc7695000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005893c1820> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon
>>>> prio=5
>>>> os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
>>>> [0x00007fcbc7796000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005893c1820> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>>>> tid=0x00007fcbf995d800 nid=0x432a waiting on condition
>>>> [0x00007fcbc7d9c000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>> at
>>>> 
>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>> at
>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
>>>> nid=0x4318 runnable [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
>>>> nid=0x4317 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
>>>> nid=0x4316 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
>>>> nid=0x4315 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
>>>> nid=0x4314 runnable [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>>>> tid=0x00007fcbf8103800 nid=0x4313 waiting on condition
>>>> [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312
>>>> in
>>>> Object.wait() [0x00007fcbe1be7000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>> - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>> 
>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
>>>> nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>> - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>>>> 
>>>> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
>>>> condition [0x00007fcbff9b9000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000589585398> (a
>>>> java.util.concurrent.CountDownLatch$Sync)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>> at
>>>> 
>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>>> at
>>>> 
>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>> at
>>>> 
>>>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>>> at scala.concurrent.Await$.ready(package.scala:86)
>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>>> at
>>>> 
>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>>> at
>>>> 
>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>> at
>>>> 
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>> at
>>>> 
>>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>>> 
>>>> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>>>> 
>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
>>>> nid=0x430b runnable
>>>> 
>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
>>>> nid=0x430c runnable
>>>> 
>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
>>>> nid=0x430d runnable
>>>> 
>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
>>>> nid=0x430e runnable
>>>> 
>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
>>>> nid=0x430f runnable
>>>> 
>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
>>>> waiting on condition
>>>> 
>>>> JNI global references: 375
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> And the final worker, no active tasks:
>>>> 
>>>> 
>>>> 2015-03-27 02:13:26
>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>> mode):
>>>> 
>>>> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
>>>> nid=0x3c5b waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
>>>> os_prio=0
>>>> tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
>>>> [0x00007f36d2e33000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005895676e8> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
>>>> nid=0x3b88 runnable [0x00007f36dafb9000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058a86a7d8> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
>>>> nid=0x3b87 runnable [0x00007f36d7138000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058abea6c8> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
>>>> nid=0x3b86 runnable [0x00007f36daeb8000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058a86a898> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
>>>> nid=0x3b85 runnable [0x00007f36d7e3b000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058abea788> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
>>>> tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition
>>>> [0x00007f36d7a39000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
>>>> waiting on condition [0x00007f36d7b3a000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x000000058aecd008> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
>>>> runnable [0x00007f36db4be000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>> at
>>>> 
>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>> at
>>>> 
>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
>>>> tid=0x00007f36f8025000
>>>> nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
>>>> nid=0x3a96 runnable [0x00007f36dc0ea000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x0000000589202c58> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
>>>> nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>>> 
>>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
>>>> nid=0x3a6c runnable [0x00007f36dcb14000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058a825c18> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
>>>> tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition
>>>> [0x00007f36dce17000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005897334d0> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
>>>> tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>> - locked <0x00000005896b0d20> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> 
>>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
>>>> tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>> - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> 
>>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
>>>> tid=0x00007f36f03a1000
>>>> nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>> 
>>>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>>> at
>>>> 
>>>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
>>>> tid=0x00007f370d9b2000
>>>> nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005895675a8> (a
>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
>>>> tid=0x00007f36f8713000
>>>> nid=0x3a63 runnable [0x00007f36da4b0000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
>>>> nid=0x3a62 runnable [0x00007f36da5b1000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
>>>> nid=0x3a61 runnable [0x00007f36da6b2000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
>>>> nid=0x3a60 runnable [0x00007f36da7b3000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
>>>> nid=0x3a5f runnable [0x00007f36da8b4000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
>>>> nid=0x3a5e runnable [0x00007f36da9b5000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon
>>>> prio=5
>>>> os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
>>>> [0x00007f36dacb6000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000589567648> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon
>>>> prio=5
>>>> os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
>>>> [0x00007f36dadb7000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000589567648> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>>>> tid=0x00007f370d97c000 nid=0x3a56 waiting on condition
>>>> [0x00007f36db3bd000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>> at
>>>> 
>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>> at
>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
>>>> nid=0x3a43 runnable [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
>>>> nid=0x3a42 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
>>>> nid=0x3a41 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
>>>> nid=0x3a40 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
>>>> nid=0x3a3f runnable [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>>>> tid=0x00007f370c101800 nid=0x3a3e waiting on condition
>>>> [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d
>>>> in
>>>> Object.wait() [0x00007f36f51e5000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>> - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>> 
>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
>>>> nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>> - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>>>> 
>>>> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
>>>> condition [0x00007f3712fda000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x000000058a974ab8> (a
>>>> java.util.concurrent.CountDownLatch$Sync)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>> at
>>>> 
>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>>> at
>>>> 
>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>> at
>>>> 
>>>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>>> at scala.concurrent.Await$.ready(package.scala:86)
>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>>> at
>>>> 
>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>>> at
>>>> 
>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>> at
>>>> 
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>> at
>>>> 
>>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>>> at
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>>> 
>>>> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>>>> 
>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
>>>> nid=0x3a34 runnable
>>>> 
>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
>>>> nid=0x3a35 runnable
>>>> 
>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
>>>> nid=0x3a36 runnable
>>>> 
>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
>>>> nid=0x3a37 runnable
>>>> 
>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
>>>> nid=0x3a3a runnable
>>>> 
>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
>>>> waiting on condition
>>>> 
>>>> JNI global references: 486
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> And the driver:
>>>> 
>>>> 2015-03-27 02:20:47
>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>> mode):
>>>> 
>>>> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
>>>> nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
>>>> nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
>>>> nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
>>>> nid=0x41d0 waiting on condition [0x00007fd38feef000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
>>>> tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition
>>>> [0x00007fd38fff0000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000fd545eb8> (a
>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5
>>>> os_prio=0
>>>> tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition
>>>> [0x00007fd3b8400000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
>>>> nid=0x417f waiting on condition [0x00007fd398935000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "task-result-getter-3" #107 daemon prio=5 os_prio=0
>>>> tid=0x00007fd3b003f800
>>>> nid=0x409c waiting on condition [0x00007fd3902f1000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "task-result-getter-2" #106 daemon prio=5 os_prio=0
>>>> tid=0x00007fd3b003d000
>>>> nid=0x409b waiting on condition [0x00007fd3903f2000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "task-result-getter-1" #105 daemon prio=5 os_prio=0
>>>> tid=0x00007fd3b003b800
>>>> nid=0x409a waiting on condition [0x00007fd3904f3000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "task-result-getter-0" #104 daemon prio=5 os_prio=0
>>>> tid=0x00007fd3b0030800
>>>> nid=0x4098 waiting on condition [0x00007fd3905f4000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
>>>> tid=0x00007fd3cc00c800
>>>> nid=0x4093 waiting on condition [0x00007fd3936f8000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
>>>> nid=0x4092 runnable [0x00007fd398b37000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e685f320> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
>>>> nid=0x4091 runnable [0x00007fd398a36000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e6871c30> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
>>>> nid=0x4090 runnable [0x00007fd39903a000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e6873520> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
>>>> tid=0x00007fd3e0011800 nid=0x4056 waiting on condition
>>>> [0x00007fd398733000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
>>>> tid=0x00007fd3dc001800 nid=0x4055 waiting on condition
>>>> [0x00007fd398834000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e7f19070> (a
>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5
>>>> os_prio=0
>>>> tid=0x00007fd3a400d800 nid=0x4051 waiting on condition
>>>> [0x00007fd398e38000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
>>>> tid=0x00007fd432c05000 nid=0x404f waiting on condition
>>>> [0x00007fd398f39000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e66fe220> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> 
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
>>>> tid=0x00007fd43297c800 nid=0x4042 waiting on condition
>>>> [0x00007fd39973b000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e740fa58> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>>> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
>>>> tid=0x00007fd4329ac000 nid=0x4041 waiting on condition
>>>> [0x00007fd39983c000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e740fce8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
>>>> tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>> - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
>>>> tid=0x00007fd4328e6800 nid=0x403b waiting on condition
>>>> [0x00007fd399c3e000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e737a0c8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>>> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
>>>> tid=0x00007fd4328eb800 nid=0x403a waiting on condition
>>>> [0x00007fd399d3f000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e737add8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
>>>> tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>> - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
>>>> tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>> - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
>>>> at
>>>> 
>>>> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>>>> 
>>>> "ResponseProcessor for block
>>>> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
>>>> prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
>>>> [0x00007fd39a941000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
>>>> at
>>>> 
>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>>> at
>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>>>> at
>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>>>> at
>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>>>> at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>>> at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>>> at
>>>> 
>>>> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
>>>> at
>>>> 
>>>> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
>>>> at
>>>> 
>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>>>> 
>>>> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
>>>> nid=0x3fda waiting on condition [0x00007fd39aa42000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e58145f0> (a
>>>> java.util.concurrent.Semaphore$NonfairSync)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
>>>> at
>>>> 
>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
>>>> at
>>>> 
>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>>> at
>>>> 
>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>>> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>>> at
>>>> 
>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>>>> 
>>>> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0
>>>> tid=0x00007fd43258c000
>>>> nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>> - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
>>>> at
>>>> 
>>>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
>>>> at
>>>> 
>>>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>>> at
>>>> 
>>>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>>> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>>> at
>>>> 
>>>> org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
>>>> at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>>>> 
>>>> "DataStreamer for file
>>>> /spark-logs/application_1427416187821_0004.inprogress
>>>> block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84
>>>> daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in
>>>> Object.wait()
>>>> [0x00007fd39ae44000]
>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at
>>>> 
>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
>>>> - locked <0x00000000e6994f30> (a java.util.LinkedList)
>>>> 
>>>> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
>>>> tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition
>>>> [0x00007fd39af45000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e6995e88> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
>>>> nid=0x3fb7 runnable [0x00007fd39b246000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e6886b58> (a
>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>> - locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>> at
>>>> 
>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Yarn application state monitor" #73 daemon prio=5 os_prio=0
>>>> tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition
>>>> [0x00007fd39b8fb000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>> 
>>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>>>> 
>>>> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
>>>> nid=0x3f77 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800
>>>> nid=0x3e6f
>>>> waiting on condition [0x00007fd39b547000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e677ae98> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>> at
>>>> 
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
>>>> tid=0x00007fd432896800 nid=0x3e6a waiting on condition
>>>> [0x00007fd39b9fc000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
>>>> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>>>> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000
>>>> nid=0x3e5e
>>>> runnable [0x00007fd3b80ff000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>> at
>>>> 
>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>> at
>>>> 
>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop"
>>>> #62
>>>> daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in
>>>> Object.wait()
>>>> [0x00007fd3b8501000]
>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
>>>> - locked <0x00000000e66af0f8> (a
>>>> org.apache.hadoop.ipc.Client$Connection)
>>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>>>> 
>>>> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
>>>> tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition
>>>> [0x00007fd3b8602000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e66e9230> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
>>>> at
>>>> 
>>>> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>>>> 
>>>> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f
>>>> in
>>>> Object.wait() [0x00007fd3b8703000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>> - locked <0x00000000e66e8190> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> 
>>>> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
>>>> nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54
>>>> daemon
>>>> prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
>>>> [0x00007fd3b8fb6000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>> at
>>>> 
>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>>>> at
>>>> 
>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>>>> - locked <0x00000000e671a620> (a java.lang.Object)
>>>> at
>>>> 
>>>> org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>>>> at
>>>> 
>>>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
>>>> tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>>>> at
>>>> 
>>>> org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
>>>> tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>> - locked <0x00000000e66609f0> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> 
>>>> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
>>>> nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
>>>> nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
>>>> nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
>>>> nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
>>>> nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
>>>> nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
>>>> nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon
>>>> prio=5
>>>> os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable
>>>> [0x00007fd3ba0de000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>> at
>>>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>>> at java.net.ServerSocket.accept(ServerSocket.java:513)
>>>> at
>>>> 
>>>> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>>> at
>>>> 
>>>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
>>>> tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>> - locked <0x00000000e6601270> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> 
>>>> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
>>>> tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>> - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> 
>>>> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
>>>> tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>> - locked <0x00000000e65380d8> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> 
>>>> "New I/O server boss #6" #36 daemon prio=5 os_prio=0
>>>> tid=0x00007fd39c04b800
>>>> nid=0x3df8 runnable [0x00007fd3ba6e2000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
>>>> nid=0x3df7 runnable [0x00007fd3ba7e3000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
>>>> nid=0x3df6 runnable [0x00007fd3ba8e4000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
>>>> nid=0x3df5 runnable [0x00007fd3ba9e5000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
>>>> nid=0x3df4 runnable [0x00007fd3baae6000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
>>>> nid=0x3df3 runnable [0x00007fd3babe7000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>> - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
>>>> - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>> at
>>>> 
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> 
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>> 
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
>>>> os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
>>>> [0x00007fd3baef9000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e64bbb40> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
>>>> os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
>>>> [0x00007fd3baffa000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e64bbb40> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5
>>>> os_prio=0
>>>> tid=0x00007fd431f99000 nid=0x3de0 waiting on condition
>>>> [0x00007fd3bb0fb000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5
>>>> os_prio=0
>>>> tid=0x00007fd431f96800 nid=0x3ddf waiting on condition
>>>> [0x00007fd3bb1fc000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5
>>>> os_prio=0
>>>> tid=0x00007fd431f86000 nid=0x3dde waiting on condition
>>>> [0x00007fd3bb2fd000]
>>>>   java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>> at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> at
>>>> 
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
>>>> tid=0x00007fd431f0e800
>>>> nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>> at
>>>> 
>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>> at
>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
>>>> nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
>>>> nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
>>>> nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
>>>> nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
>>>> nid=0x3d72 waiting on condition [0x00007fd40c460000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
>>>> nid=0x3d71 waiting on condition [0x00007fd40c561000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
>>>> nid=0x3d70 waiting on condition [0x00007fd40c662000]
>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon
>>>> prio=5
>>>> os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable
>>>> [0x00007fd40c966000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>> at
>>>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>>> at java.net.ServerSocket.accept(ServerSocket.java:513)
>>>> at
>>>> 
>>>> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>>> at
>>>> 
>>>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>> at
>>>> 
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 
>>>> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
>>>> nid=0x3d67 runnable [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
>>>> nid=0x3d66 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
>>>> nid=0x3d65 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
>>>> nid=0x3d64 waiting on condition [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
>>>> nid=0x3d63 runnable [0x0000000000000000]
>>>>   java.lang.Thread.State: RUNNABLE
>>>> 
>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62
>>>> in
>>>> Object.wait() [0x00007fd40f28e000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>> - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>> 
>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
>>>> nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>> - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>>>> 
>>>> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
>>>> Object.wait() [0x00007fd438220000]
>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
>>>> - locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
>>>> at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
>>>> at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
>>>> at
>>>> 
>>>> org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
>>>> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
>>>> at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
>>>> at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
>>>> at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
>>>> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
>>>> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
>>>> at
>>>> 
>>>> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>>>> at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
>>>> at $line24.$read$$iwC$$iwC.<init>(<console>:35)
>>>> at $line24.$read$$iwC.<init>(<console>:37)
>>>> at $line24.$read.<init>(<console>:39)
>>>> at $line24.$read$.<init>(<console>:43)
>>>> at $line24.$read$.<clinit>(<console>)
>>>> at $line24.$eval$.<init>(<console>:7)
>>>> at $line24.$eval$.<clinit>(<console>)
>>>> at $line24.$eval.$print(<console>)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> 
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> at
>>>> 
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>> at
>>>> 
>>>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>>>> at
>>>> 
>>>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>>>> at
>>>> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>>>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>>>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>>>> at
>>>> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
>>>> at
>>>> 
>>>> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
>>>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
>>>> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
>>>> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
>>>> at
>>>> 
>>>> org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
>>>> at
>>>> 
>>>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
>>>> at
>>>> 
>>>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>>> at
>>>> 
>>>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>>> at
>>>> 
>>>> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>>>> at
>>>> 
>>>> org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>>>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
>>>> at org.apache.spark.repl.Main$.main(Main.scala:31)
>>>> at org.apache.spark.repl.Main.main(Main.scala)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> 
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> at
>>>> 
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>> at
>>>> 
>>>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>> 
>>>> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>>>> 
>>>> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000
>>>> nid=0x3d5c
>>>> runnable
>>>> 
>>>> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800
>>>> nid=0x3d5d
>>>> runnable
>>>> 
>>>> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800
>>>> nid=0x3d5e
>>>> runnable
>>>> 
>>>> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000
>>>> nid=0x3d5f
>>>> runnable
>>>> 
>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
>>>> waiting on condition
>>>> 
>>>> JNI global references: 420
>>>> 
>>>> 
>>>> 
>>>> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust
>>>> <mi...@databricks.com>
>>>> wrote:
>>>>> 
>>>>> Is it possible to jstack the executors and see where they are hanging?
>>>>> 
>>>>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:
>>>>>> 
>>>>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu,
>>>>>> 30GB),
>>>>>> executor memory 20GB, driver memory 10GB
>>>>>> 
>>>>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data
>>>>>> spread
>>>>>> out over roughly 2,000 Parquet files and my queries frequently hang.
>>>>>> Simple
>>>>>> queries like "select count(*) from ..." on the entire data set work
>>>>>> ok.
>>>>>> Slightly more demanding ones with group by's and some aggregate
>>>>>> functions
>>>>>> (percentile_approx, avg, etc.) work ok as well, as long as I have some
>>>>>> criteria in my where clause to keep the number of rows down.
>>>>>> 
>>>>>> Once I hit some limit on query complexity and rows processed, my
>>>>>> queries
>>>>>> start to hang.  I've left them for up to an hour without seeing any
>>>>>> progress.  No OOM's either - the job is just stuck.
>>>>>> 
>>>>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800,
>>>>>> but
>>>>>> with the same results: usually near the end of the tasks (like 780 of
>>>>>> 800
>>>>>> complete), progress just stops:
>>>>>> 
>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0
>>>>>> in
>>>>>> stage 1.0 (TID 1618) in 800 ms on
>>>>>> ip-10-209-22-211.eu-west-1.compute.internal (748/800)
>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0
>>>>>> in
>>>>>> stage 1.0 (TID 1623) in 622 ms on
>>>>>> ip-10-105-12-41.eu-west-1.compute.internal
>>>>>> (749/800)
>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0
>>>>>> in
>>>>>> stage 1.0 (TID 1627) in 616 ms on
>>>>>> ip-10-90-2-201.eu-west-1.compute.internal
>>>>>> (750/800)
>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0
>>>>>> in
>>>>>> stage 1.0 (TID 1629) in 611 ms on
>>>>>> ip-10-90-2-201.eu-west-1.compute.internal
>>>>>> (751/800)
>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0
>>>>>> in
>>>>>> stage 1.0 (TID 1625) in 669 ms on
>>>>>> ip-10-105-12-41.eu-west-1.compute.internal
>>>>>> (752/800)
>>>>>> 
>>>>>> ^^^^^^^ this is where it stays forever
>>>>>> 
>>>>>> Looking at the Spark UI, several of the executors still list active
>>>>>> tasks.  I do see that the Shuffle Read for executors that don't have
>>>>>> any
>>>>>> tasks remaining is around 100MB, whereas it's more like 10MB for the
>>>>>> executors that still have tasks.
>>>>>> 
>>>>>> The first stage, mapPartitions, always completes fine.  It's the
>>>>>> second
>>>>>> stage (takeOrdered), that hangs.
>>>>>> 
>>>>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
>>>>>> encountered it when using JSON files (instead of Parquet).
>>>>>> 
>>>>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
>>>>>> are having this issue.
>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Christian Perez
>>> Silicon Valley Data Science
>>> Data Analyst
>>> christian@svds.com
>>> @cp_phd
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>> 
>> 
> 
> 
> 
> -- 
> Christian Perez
> Silicon Valley Data Science
> Data Analyst
> christian@svds.com
> @cp_phd
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark SQL queries hang forever

Posted by Christian Perez <ch...@svds.com>.
I think I've seen a timeout in one of the executor logs... but I'm
usually not that patient.

If I kill the executor running the last task started on spark UI, 2
out of 3 times it will retry and complete successfully.

I've found that everything works when running in local mode versus
YARN. Have not tried Spark standalone. Something (block fetcher?) is
hanging during shuffle when distributed...

On Tue, Mar 31, 2015 at 12:10 AM, Zsolt Tóth <to...@gmail.com> wrote:
> I have a very similar problem with collect() but only on relatively large
> input data, on Amazon EMR. Look for my thread for details if interested.
> I get "connection timed out" exception after a 15-20 minutes in one of the
> container logs - does it happen in your case?
>
> 2015-03-30 23:34 GMT+02:00 Christian Perez <ch...@svds.com>:
>>
>> I'm having the exact same problem (hanging) with takeOrdered (using
>> PySpark's orderBy on a DataFrame.)
>>
>> On Thu, Mar 26, 2015 at 7:21 PM, Jon Chase <jo...@gmail.com> wrote:
>> > Here are the jstacks from a 3 worker cluster.  Two of the workers are
>> > sitting with 4 active tasks, the other has no active tasks.  Not sure if
>> > it
>> > makes a difference, but I only see an ExecutorLauncher instance via JPS
>> > on
>> > the worker that has no active tasks.
>> >
>> > Here's the first stack dump, this machine has 4 active tasks:
>> >
>> > 2015-03-27 02:11:46
>> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>> > mode):
>> >
>> > "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
>> > nid=0x43a3 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
>> > os_prio=0
>> > tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
>> > [0x00007f91bad9a000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x0000000589884fc0> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
>> > nid=0x42e2 runnable [0x00007f91b4c54000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000005898aa000> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
>> > nid=0x42e1 runnable [0x00007f91b5d56000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058ad5b5f8> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
>> > nid=0x42e0 runnable [0x00007f91b5e57000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000005898aa0c0> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
>> > nid=0x42df runnable [0x00007f91b5f58000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058ad5b6b8> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
>> > tid=0x00007f91d0773800 nid=0x427b waiting on condition
>> > [0x00007f91b6258000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000006cfda6800> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> > at
>> >
>> > org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> > at
>> >
>> > org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
>> > tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition
>> > [0x00007f91b6b5a000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
>> > waiting on condition [0x00007f91b6c5b000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x000000058af8e0b8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
>> > runnable [0x00007f91baf9c000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> > at
>> >
>> > org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> > at
>> >
>> > org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0
>> > tid=0x00007f91d8118800
>> > nid=0x41ee waiting on condition [0x00007f91bae9b000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> > io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> > at
>> >
>> > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
>> > nid=0x41ed runnable [0x00007f91baa97000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x0000000589224af0> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
>> > tid=0x00007f91d0019000 nid=0x41ea waiting on condition
>> > [0x00007f91bac98000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000006c8837750> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> > at
>> >
>> > org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> > at
>> >
>> > org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
>> > tid=0x00007f91d0016800 nid=0x41e9 waiting on condition
>> > [0x00007f91ba16d000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000006cc53bff0> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> > at
>> >
>> > org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> > at
>> >
>> > org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
>> > tid=0x00007f91d0015000 nid=0x41e8 waiting on condition
>> > [0x00007f91b9f6b000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000006cfda69b0> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> > at
>> >
>> > org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> > at
>> >
>> > org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
>> > nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>> >
>> > "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
>> > nid=0x41c6 runnable [0x00007f91bbac7000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058984ee60> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
>> > tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition
>> > [0x00007f91bbbc8000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000005898f7290> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
>> > tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.util.TimerThread.mainLoop(Timer.java:526)
>> > - locked <0x000000058974cb00> (a java.util.TaskQueue)
>> > at java.util.TimerThread.run(Timer.java:505)
>> >
>> > "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
>> > tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <0x0000000589904038> (a java.util.TaskQueue)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.util.TimerThread.mainLoop(Timer.java:526)
>> > - locked <0x0000000589904038> (a java.util.TaskQueue)
>> > at java.util.TimerThread.run(Timer.java:505)
>> >
>> > "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
>> > tid=0x00007f91ed92d800
>> > nid=0x41bf waiting on condition [0x00007f91bab98000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x0000000589884e80> (a
>> > scala.concurrent.forkjoin.ForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0
>> > tid=0x00007f91ed92c800
>> > nid=0x41be waiting on condition [0x00007f91ba370000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> >
>> > org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>> > at
>> >
>> > org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O server boss #12" #39 daemon prio=5 os_prio=0
>> > tid=0x0000000000a02000
>> > nid=0x41bd runnable [0x00007f91b9564000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
>> > - locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
>> > nid=0x41bc runnable [0x00007f91b9665000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
>> > - locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
>> > nid=0x41bb runnable [0x00007f91b9766000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
>> > - locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
>> > nid=0x41ba runnable [0x00007f91b9867000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
>> > - locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
>> > nid=0x41b9 runnable [0x00007f91b9968000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
>> > - locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
>> > nid=0x41b8 runnable [0x00007f91b9c69000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
>> > - locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon
>> > prio=5
>> > os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
>> > [0x00007f91b9d6a000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x0000000589884f20> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon
>> > prio=5
>> > os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
>> > [0x00007f91b9e6b000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x0000000589884f20> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>> > tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> > at
>> >
>> > akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> > at
>> > akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
>> > nid=0x419e runnable [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
>> > nid=0x419d waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
>> > nid=0x419c waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
>> > nid=0x419b waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
>> > nid=0x419a runnable [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>> > tid=0x00007f91ec101000 nid=0x4199 waiting on condition
>> > [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198
>> > in
>> > Object.wait() [0x00007f91d4254000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> > - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>> >
>> > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
>> > nid=0x4197 in Object.wait() [0x00007f91d4355000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> > - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>> >
>> > "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
>> > condition [0x00007f91f1fdf000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x0000000589a1b5f0> (a
>> > java.util.concurrent.CountDownLatch$Sync)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>> > at
>> >
>> > akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>> > at
>> >
>> > akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> > at
>> >
>> > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> > at scala.concurrent.Await$.ready(package.scala:86)
>> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>> > at
>> >
>> > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>> > at
>> >
>> > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:422)
>> > at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> > at
>> >
>> > org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>> >
>> > "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>> >
>> > "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
>> > nid=0x4191 runnable
>> >
>> > "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
>> > nid=0x4192 runnable
>> >
>> > "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
>> > nid=0x4193 runnable
>> >
>> > "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
>> > nid=0x4194 runnable
>> >
>> > "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
>> > nid=0x4195 runnable
>> >
>> > "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
>> > waiting on condition
>> >
>> > JNI global references: 319
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Here's the next worker, also for active tasks:
>> >
>> > 2015-03-27 02:12:54
>> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>> > mode):
>> >
>> > "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
>> > nid=0x44f1 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
>> > os_prio=0
>> > tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
>> > [0x00007fcbbf87f000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x0000000589510428> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
>> > tid=0x00007fcbe4028800 nid=0x445f waiting on condition
>> > [0x00007fcbbf97f000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000006126cd358> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> > at
>> >
>> > org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> > at
>> >
>> > org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
>> > nid=0x4459 runnable [0x00007fcbc88c7000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058a6b8c98> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
>> > nid=0x4458 runnable [0x00007fcbc7897000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058a6e5a18> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
>> > nid=0x4457 runnable [0x00007fcbc7998000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058aade110> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
>> > nid=0x4456 runnable [0x00007fcbc7b9a000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058aade1d0> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
>> > tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition
>> > [0x00007fcbc4886000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x0000000615539dd8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> > at
>> >
>> > org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> > at
>> >
>> > org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
>> > tid=0x00007fcbe4165000 nid=0x4453 waiting on condition
>> > [0x00007fcbca620000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x0000000617bd4238> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> > at
>> >
>> > org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> > at
>> >
>> > org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
>> > tid=0x00007fcbe4164000 nid=0x4452 waiting on condition
>> > [0x00007fcbc87c5000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000006126cd538> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> > at
>> >
>> > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> > at
>> >
>> > org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> > at
>> >
>> > org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> > at
>> >
>> > org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at
>> > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
>> > tid=0x00007fcbdcff1800 nid=0x436e waiting on condition
>> > [0x00007fcbc4485000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
>> > waiting on condition [0x00007fcbc4586000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x000000058adabe70> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
>> > runnable [0x00007fcbc7e9d000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> > at
>> >
>> > org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> > at
>> >
>> > org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
>> > tid=0x00007fcbf9608800
>> > nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> > io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> > at
>> >
>> > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
>> > nid=0x4364 runnable [0x00007fcbc8ac9000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x0000000589116ae0> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
>> > nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>> >
>> > "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
>> > nid=0x4340 runnable [0x00007fcbc94f3000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058964c068> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
>> > tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition
>> > [0x00007fcbc97f6000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x000000058a6fa058> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
>> > tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <0x000000058928c438> (a java.util.TaskQueue)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.util.TimerThread.mainLoop(Timer.java:526)
>> > - locked <0x000000058928c438> (a java.util.TaskQueue)
>> > at java.util.TimerThread.run(Timer.java:505)
>> >
>> > "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
>> > tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.util.TimerThread.mainLoop(Timer.java:526)
>> > - locked <0x000000058928c5a0> (a java.util.TaskQueue)
>> > at java.util.TimerThread.run(Timer.java:505)
>> >
>> > "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
>> > tid=0x00007fcbdc29d000
>> > nid=0x433a waiting on condition [0x00007fcbc85c4000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> >
>> > org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>> > at
>> >
>> > org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
>> > tid=0x00007fcbe4474800
>> > nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x000000058a70a468> (a
>> > scala.concurrent.forkjoin.ForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "New I/O server boss #12" #41 daemon prio=5 os_prio=0
>> > tid=0x00007fcbdc29a000
>> > nid=0x4337 runnable [0x00007fcbc6e8f000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
>> > - locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
>> > nid=0x4336 runnable [0x00007fcbc6f90000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
>> > - locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
>> > nid=0x4335 runnable [0x00007fcbc7091000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
>> > - locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
>> > nid=0x4334 runnable [0x00007fcbc7392000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
>> > - locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
>> > nid=0x4333 runnable [0x00007fcbc7493000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
>> > - locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
>> > nid=0x4332 runnable [0x00007fcbc7594000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
>> > - locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon
>> > prio=5
>> > os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
>> > [0x00007fcbc7695000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000005893c1820> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon
>> > prio=5
>> > os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
>> > [0x00007fcbc7796000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000005893c1820> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>> > tid=0x00007fcbf995d800 nid=0x432a waiting on condition
>> > [0x00007fcbc7d9c000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> > at
>> >
>> > akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> > at
>> > akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
>> > nid=0x4318 runnable [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
>> > nid=0x4317 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
>> > nid=0x4316 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
>> > nid=0x4315 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
>> > nid=0x4314 runnable [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>> > tid=0x00007fcbf8103800 nid=0x4313 waiting on condition
>> > [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312
>> > in
>> > Object.wait() [0x00007fcbe1be7000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> > - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>> >
>> > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
>> > nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> > - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>> >
>> > "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
>> > condition [0x00007fcbff9b9000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x0000000589585398> (a
>> > java.util.concurrent.CountDownLatch$Sync)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>> > at
>> >
>> > akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>> > at
>> >
>> > akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> > at
>> >
>> > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> > at scala.concurrent.Await$.ready(package.scala:86)
>> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>> > at
>> >
>> > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>> > at
>> >
>> > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:422)
>> > at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> > at
>> >
>> > org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>> >
>> > "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>> >
>> > "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
>> > nid=0x430b runnable
>> >
>> > "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
>> > nid=0x430c runnable
>> >
>> > "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
>> > nid=0x430d runnable
>> >
>> > "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
>> > nid=0x430e runnable
>> >
>> > "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
>> > nid=0x430f runnable
>> >
>> > "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
>> > waiting on condition
>> >
>> > JNI global references: 375
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > And the final worker, no active tasks:
>> >
>> >
>> > 2015-03-27 02:13:26
>> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>> > mode):
>> >
>> > "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
>> > nid=0x3c5b waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
>> > os_prio=0
>> > tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
>> > [0x00007f36d2e33000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000005895676e8> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
>> > nid=0x3b88 runnable [0x00007f36dafb9000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058a86a7d8> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
>> > nid=0x3b87 runnable [0x00007f36d7138000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058abea6c8> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
>> > nid=0x3b86 runnable [0x00007f36daeb8000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058a86a898> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
>> > nid=0x3b85 runnable [0x00007f36d7e3b000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058abea788> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
>> > tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition
>> > [0x00007f36d7a39000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
>> > waiting on condition [0x00007f36d7b3a000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x000000058aecd008> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
>> > runnable [0x00007f36db4be000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> > at
>> >
>> > org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> > at
>> >
>> > org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
>> > tid=0x00007f36f8025000
>> > nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> > io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> > at
>> >
>> > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
>> > nid=0x3a96 runnable [0x00007f36dc0ea000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x0000000589202c58> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
>> > nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>> >
>> > "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
>> > nid=0x3a6c runnable [0x00007f36dcb14000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058a825c18> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
>> > tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition
>> > [0x00007f36dce17000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000005897334d0> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
>> > tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.util.TimerThread.mainLoop(Timer.java:526)
>> > - locked <0x00000005896b0d20> (a java.util.TaskQueue)
>> > at java.util.TimerThread.run(Timer.java:505)
>> >
>> > "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
>> > tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.util.TimerThread.mainLoop(Timer.java:526)
>> > - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
>> > at java.util.TimerThread.run(Timer.java:505)
>> >
>> > "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
>> > tid=0x00007f36f03a1000
>> > nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> >
>> > org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>> > at
>> >
>> > org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
>> > tid=0x00007f370d9b2000
>> > nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000005895675a8> (a
>> > scala.concurrent.forkjoin.ForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "New I/O server boss #12" #41 daemon prio=5 os_prio=0
>> > tid=0x00007f36f8713000
>> > nid=0x3a63 runnable [0x00007f36da4b0000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
>> > - locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
>> > nid=0x3a62 runnable [0x00007f36da5b1000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
>> > - locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
>> > nid=0x3a61 runnable [0x00007f36da6b2000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
>> > - locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
>> > nid=0x3a60 runnable [0x00007f36da7b3000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
>> > - locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
>> > nid=0x3a5f runnable [0x00007f36da8b4000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
>> > - locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
>> > nid=0x3a5e runnable [0x00007f36da9b5000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
>> > - locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon
>> > prio=5
>> > os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
>> > [0x00007f36dacb6000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x0000000589567648> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon
>> > prio=5
>> > os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
>> > [0x00007f36dadb7000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x0000000589567648> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>> > tid=0x00007f370d97c000 nid=0x3a56 waiting on condition
>> > [0x00007f36db3bd000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> > at
>> >
>> > akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> > at
>> > akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
>> > nid=0x3a43 runnable [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
>> > nid=0x3a42 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
>> > nid=0x3a41 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
>> > nid=0x3a40 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
>> > nid=0x3a3f runnable [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>> > tid=0x00007f370c101800 nid=0x3a3e waiting on condition
>> > [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d
>> > in
>> > Object.wait() [0x00007f36f51e5000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> > - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>> >
>> > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
>> > nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> > - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>> >
>> > "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
>> > condition [0x00007f3712fda000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x000000058a974ab8> (a
>> > java.util.concurrent.CountDownLatch$Sync)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>> > at
>> >
>> > akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>> > at
>> >
>> > akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> > at
>> >
>> > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> > at scala.concurrent.Await$.ready(package.scala:86)
>> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>> > at
>> >
>> > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>> > at
>> >
>> > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:422)
>> > at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> > at
>> >
>> > org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>> > at
>> >
>> > org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>> >
>> > "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>> >
>> > "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
>> > nid=0x3a34 runnable
>> >
>> > "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
>> > nid=0x3a35 runnable
>> >
>> > "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
>> > nid=0x3a36 runnable
>> >
>> > "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
>> > nid=0x3a37 runnable
>> >
>> > "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
>> > nid=0x3a3a runnable
>> >
>> > "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
>> > waiting on condition
>> >
>> > JNI global references: 486
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > And the driver:
>> >
>> > 2015-03-27 02:20:47
>> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>> > mode):
>> >
>> > "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
>> > nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e67194f0> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
>> > nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e67194f0> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
>> > nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e67194f0> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
>> > nid=0x41d0 waiting on condition [0x00007fd38feef000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e67194f0> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
>> > tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition
>> > [0x00007fd38fff0000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000fd545eb8> (a
>> > scala.concurrent.forkjoin.ForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5
>> > os_prio=0
>> > tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition
>> > [0x00007fd3b8400000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e64f49f8> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
>> > nid=0x417f waiting on condition [0x00007fd398935000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e67194f0> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "task-result-getter-3" #107 daemon prio=5 os_prio=0
>> > tid=0x00007fd3b003f800
>> > nid=0x409c waiting on condition [0x00007fd3902f1000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e66e9a00> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "task-result-getter-2" #106 daemon prio=5 os_prio=0
>> > tid=0x00007fd3b003d000
>> > nid=0x409b waiting on condition [0x00007fd3903f2000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e66e9a00> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "task-result-getter-1" #105 daemon prio=5 os_prio=0
>> > tid=0x00007fd3b003b800
>> > nid=0x409a waiting on condition [0x00007fd3904f3000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e66e9a00> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "task-result-getter-0" #104 daemon prio=5 os_prio=0
>> > tid=0x00007fd3b0030800
>> > nid=0x4098 waiting on condition [0x00007fd3905f4000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e66e9a00> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
>> > tid=0x00007fd3cc00c800
>> > nid=0x4093 waiting on condition [0x00007fd3936f8000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> > io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> > at
>> >
>> > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
>> > nid=0x4092 runnable [0x00007fd398b37000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e685f320> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
>> > nid=0x4091 runnable [0x00007fd398a36000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e6871c30> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
>> > nid=0x4090 runnable [0x00007fd39903a000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e6873520> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
>> > tid=0x00007fd3e0011800 nid=0x4056 waiting on condition
>> > [0x00007fd398733000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
>> > tid=0x00007fd3dc001800 nid=0x4055 waiting on condition
>> > [0x00007fd398834000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e7f19070> (a
>> > scala.concurrent.forkjoin.ForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5
>> > os_prio=0
>> > tid=0x00007fd3a400d800 nid=0x4051 waiting on condition
>> > [0x00007fd398e38000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e64f49f8> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
>> > tid=0x00007fd432c05000 nid=0x404f waiting on condition
>> > [0x00007fd398f39000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e66fe220> (a
>> > java.util.concurrent.SynchronousQueue$TransferStack)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> > at
>> >
>> > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
>> > tid=0x00007fd43297c800 nid=0x4042 waiting on condition
>> > [0x00007fd39973b000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e740fa58> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>> > at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
>> > tid=0x00007fd4329ac000 nid=0x4041 waiting on condition
>> > [0x00007fd39983c000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e740fce8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
>> > tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> > - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> > at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
>> > tid=0x00007fd4328e6800 nid=0x403b waiting on condition
>> > [0x00007fd399c3e000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e737a0c8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>> > at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
>> > tid=0x00007fd4328eb800 nid=0x403a waiting on condition
>> > [0x00007fd399d3f000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e737add8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
>> > tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> > - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> > at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
>> > tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>> >    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> > - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
>> > at
>> >
>> > com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>> >
>> > "ResponseProcessor for block
>> > BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
>> > prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
>> > [0x00007fd39a941000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
>> > - locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
>> > at
>> >
>> > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>> > at
>> > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>> > at
>> > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>> > at
>> > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>> > at java.io.FilterInputStream.read(FilterInputStream.java:83)
>> > at java.io.FilterInputStream.read(FilterInputStream.java:83)
>> > at
>> >
>> > org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
>> > at
>> >
>> > org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
>> > at
>> >
>> > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>> >
>> > "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
>> > nid=0x3fda waiting on condition [0x00007fd39aa42000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e58145f0> (a
>> > java.util.concurrent.Semaphore$NonfairSync)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> > at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
>> > at
>> >
>> > org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
>> > at
>> >
>> > org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>> > at
>> >
>> > org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>> > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>> > at
>> >
>> > org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>> >
>> > "Spark Context Cleaner" #85 daemon prio=5 os_prio=0
>> > tid=0x00007fd43258c000
>> > nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>> >    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> > - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
>> > at
>> >
>> > org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
>> > at
>> >
>> > org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>> > at
>> >
>> > org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>> > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>> > at
>> >
>> > org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
>> > at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>> >
>> > "DataStreamer for file
>> > /spark-logs/application_1427416187821_0004.inprogress
>> > block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84
>> > daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in
>> > Object.wait()
>> > [0x00007fd39ae44000]
>> >    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at
>> >
>> > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
>> > - locked <0x00000000e6994f30> (a java.util.LinkedList)
>> >
>> > "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
>> > tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition
>> > [0x00007fd39af45000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e6995e88> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
>> > nid=0x3fb7 runnable [0x00007fd39b246000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e6886b58> (a
>> > io.netty.channel.nio.SelectedSelectionKeySet)
>> > - locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> > at
>> >
>> > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Yarn application state monitor" #73 daemon prio=5 os_prio=0
>> > tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition
>> > [0x00007fd39b8fb000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at
>> >
>> > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>> >
>> > "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
>> > nid=0x3f77 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800
>> > nid=0x3e6f
>> > waiting on condition [0x00007fd39b547000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e677ae98> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
>> > tid=0x00007fd432896800 nid=0x3e6a waiting on condition
>> > [0x00007fd39b9fc000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
>> > at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>> > at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000
>> > nid=0x3e5e
>> > runnable [0x00007fd3b80ff000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> > at
>> >
>> > org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> > at
>> >
>> > org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop"
>> > #62
>> > daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in
>> > Object.wait()
>> > [0x00007fd3b8501000]
>> >    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
>> > - locked <0x00000000e66af0f8> (a
>> > org.apache.hadoop.ipc.Client$Connection)
>> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>> >
>> > "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
>> > tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition
>> > [0x00007fd3b8602000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e66e9230> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
>> > at
>> >
>> > java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
>> > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>> >
>> > "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f
>> > in
>> > Object.wait() [0x00007fd3b8703000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.util.TimerThread.mainLoop(Timer.java:526)
>> > - locked <0x00000000e66e8190> (a java.util.TaskQueue)
>> > at java.util.TimerThread.run(Timer.java:505)
>> >
>> > "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
>> > nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e67194f0> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54
>> > daemon
>> > prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
>> > [0x00007fd3b8fb6000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>> > at
>> >
>> > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>> > at
>> >
>> > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>> > - locked <0x00000000e671a620> (a java.lang.Object)
>> > at
>> >
>> > org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>> > at
>> >
>> > org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
>> > tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
>> > - locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>> > at
>> >
>> > org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
>> > tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.util.TimerThread.mainLoop(Timer.java:526)
>> > - locked <0x00000000e66609f0> (a java.util.TaskQueue)
>> > at java.util.TimerThread.run(Timer.java:505)
>> >
>> > "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
>> > nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e6539dc8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
>> > nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e6539dc8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
>> > nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e6539dc8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
>> > nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e6539dc8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
>> > nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e6539dc8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
>> > nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e6539dc8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
>> > nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e6539dc8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon
>> > prio=5
>> > os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable
>> > [0x00007fd3ba0de000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at java.net.PlainSocketImpl.socketAccept(Native Method)
>> > at
>> > java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>> > at java.net.ServerSocket.accept(ServerSocket.java:513)
>> > at
>> >
>> > org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>> > at
>> >
>> > org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
>> > tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.util.TimerThread.mainLoop(Timer.java:526)
>> > - locked <0x00000000e6601270> (a java.util.TaskQueue)
>> > at java.util.TimerThread.run(Timer.java:505)
>> >
>> > "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
>> > tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.util.TimerThread.mainLoop(Timer.java:526)
>> > - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
>> > at java.util.TimerThread.run(Timer.java:505)
>> >
>> > "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
>> > tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.util.TimerThread.mainLoop(Timer.java:526)
>> > - locked <0x00000000e65380d8> (a java.util.TaskQueue)
>> > at java.util.TimerThread.run(Timer.java:505)
>> >
>> > "New I/O server boss #6" #36 daemon prio=5 os_prio=0
>> > tid=0x00007fd39c04b800
>> > nid=0x3df8 runnable [0x00007fd3ba6e2000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
>> > - locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
>> > nid=0x3df7 runnable [0x00007fd3ba7e3000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
>> > - locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
>> > nid=0x3df6 runnable [0x00007fd3ba8e4000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
>> > - locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
>> > nid=0x3df5 runnable [0x00007fd3ba9e5000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
>> > - locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
>> > nid=0x3df4 runnable [0x00007fd3baae6000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
>> > - locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
>> > nid=0x3df3 runnable [0x00007fd3babe7000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> > - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
>> > - locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
>> > - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
>> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> > at
>> >
>> > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> > at
>> >
>> > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> > at
>> >
>> > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
>> > os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
>> > [0x00007fd3baef9000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e64bbb40> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
>> > os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
>> > [0x00007fd3baffa000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e64bbb40> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5
>> > os_prio=0
>> > tid=0x00007fd431f99000 nid=0x3de0 waiting on condition
>> > [0x00007fd3bb0fb000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e64f49f8> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5
>> > os_prio=0
>> > tid=0x00007fd431f96800 nid=0x3ddf waiting on condition
>> > [0x00007fd3bb1fc000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e64f49f8> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5
>> > os_prio=0
>> > tid=0x00007fd431f86000 nid=0x3dde waiting on condition
>> > [0x00007fd3bb2fd000]
>> >    java.lang.Thread.State: WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e64f49f8> (a
>> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> > at
>> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> > at
>> >
>> > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> > "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
>> > tid=0x00007fd431f0e800
>> > nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> > at java.lang.Thread.sleep(Native Method)
>> > at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> > at
>> >
>> > akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> > at
>> > akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
>> > nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e42808f8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
>> > nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e42808f8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
>> > nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e42808f8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
>> > nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e42808f8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
>> > nid=0x3d72 waiting on condition [0x00007fd40c460000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e42808f8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
>> > nid=0x3d71 waiting on condition [0x00007fd40c561000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e42808f8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
>> > nid=0x3d70 waiting on condition [0x00007fd40c662000]
>> >    java.lang.Thread.State: TIMED_WAITING (parking)
>> > at sun.misc.Unsafe.park(Native Method)
>> > - parking to wait for  <0x00000000e42808f8> (a
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > at
>> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> > at
>> >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> > at
>> >
>> > org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon
>> > prio=5
>> > os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable
>> > [0x00007fd40c966000]
>> >    java.lang.Thread.State: RUNNABLE
>> > at java.net.PlainSocketImpl.socketAccept(Native Method)
>> > at
>> > java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>> > at java.net.ServerSocket.accept(ServerSocket.java:513)
>> > at
>> >
>> > org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>> > at
>> >
>> > org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> > at
>> >
>> > org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
>> > nid=0x3d67 runnable [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
>> > nid=0x3d66 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
>> > nid=0x3d65 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
>> > nid=0x3d64 waiting on condition [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
>> > nid=0x3d63 runnable [0x0000000000000000]
>> >    java.lang.Thread.State: RUNNABLE
>> >
>> > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62
>> > in
>> > Object.wait() [0x00007fd40f28e000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> > - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
>> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>> >
>> > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
>> > nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.Object.wait(Object.java:502)
>> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> > - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>> >
>> > "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
>> > Object.wait() [0x00007fd438220000]
>> >    java.lang.Thread.State: WAITING (on object monitor)
>> > at java.lang.Object.wait(Native Method)
>> > at java.lang.Object.wait(Object.java:502)
>> > at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
>> > - locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
>> > at
>> > org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
>> > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
>> > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
>> > at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
>> > at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
>> > at
>> >
>> > org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
>> > at
>> >
>> > org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
>> > at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
>> > at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
>> > at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
>> > at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
>> > at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
>> > at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
>> > at
>> >
>> > $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
>> > at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
>> > at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>> > at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>> > at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>> > at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
>> > at $line24.$read$$iwC$$iwC.<init>(<console>:35)
>> > at $line24.$read$$iwC.<init>(<console>:37)
>> > at $line24.$read.<init>(<console>:39)
>> > at $line24.$read$.<init>(<console>:43)
>> > at $line24.$read$.<clinit>(<console>)
>> > at $line24.$eval$.<init>(<console>:7)
>> > at $line24.$eval$.<clinit>(<console>)
>> > at $line24.$eval.$print(<console>)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > at
>> >
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:497)
>> > at
>> >
>> > org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>> > at
>> >
>> > org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>> > at
>> > org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>> > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>> > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>> > at
>> > org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
>> > at
>> >
>> > org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
>> > at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
>> > at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
>> > at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
>> > at
>> >
>> > org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
>> > at
>> >
>> > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
>> > at
>> >
>> > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>> > at
>> >
>> > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>> > at
>> >
>> > scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>> > at
>> >
>> > org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>> > at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
>> > at org.apache.spark.repl.Main$.main(Main.scala:31)
>> > at org.apache.spark.repl.Main.main(Main.scala)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > at
>> >
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:497)
>> > at
>> >
>> > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>> > at
>> > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>> > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>> > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>> > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> >
>> > "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>> >
>> > "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000
>> > nid=0x3d5c
>> > runnable
>> >
>> > "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800
>> > nid=0x3d5d
>> > runnable
>> >
>> > "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800
>> > nid=0x3d5e
>> > runnable
>> >
>> > "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000
>> > nid=0x3d5f
>> > runnable
>> >
>> > "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
>> > waiting on condition
>> >
>> > JNI global references: 420
>> >
>> >
>> >
>> > On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust
>> > <mi...@databricks.com>
>> > wrote:
>> >>
>> >> Is it possible to jstack the executors and see where they are hanging?
>> >>
>> >> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:
>> >>>
>> >>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu,
>> >>> 30GB),
>> >>> executor memory 20GB, driver memory 10GB
>> >>>
>> >>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data
>> >>> spread
>> >>> out over roughly 2,000 Parquet files and my queries frequently hang.
>> >>> Simple
>> >>> queries like "select count(*) from ..." on the entire data set work
>> >>> ok.
>> >>> Slightly more demanding ones with group by's and some aggregate
>> >>> functions
>> >>> (percentile_approx, avg, etc.) work ok as well, as long as I have some
>> >>> criteria in my where clause to keep the number of rows down.
>> >>>
>> >>> Once I hit some limit on query complexity and rows processed, my
>> >>> queries
>> >>> start to hang.  I've left them for up to an hour without seeing any
>> >>> progress.  No OOM's either - the job is just stuck.
>> >>>
>> >>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800,
>> >>> but
>> >>> with the same results: usually near the end of the tasks (like 780 of
>> >>> 800
>> >>> complete), progress just stops:
>> >>>
>> >>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0
>> >>> in
>> >>> stage 1.0 (TID 1618) in 800 ms on
>> >>> ip-10-209-22-211.eu-west-1.compute.internal (748/800)
>> >>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0
>> >>> in
>> >>> stage 1.0 (TID 1623) in 622 ms on
>> >>> ip-10-105-12-41.eu-west-1.compute.internal
>> >>> (749/800)
>> >>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0
>> >>> in
>> >>> stage 1.0 (TID 1627) in 616 ms on
>> >>> ip-10-90-2-201.eu-west-1.compute.internal
>> >>> (750/800)
>> >>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0
>> >>> in
>> >>> stage 1.0 (TID 1629) in 611 ms on
>> >>> ip-10-90-2-201.eu-west-1.compute.internal
>> >>> (751/800)
>> >>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0
>> >>> in
>> >>> stage 1.0 (TID 1625) in 669 ms on
>> >>> ip-10-105-12-41.eu-west-1.compute.internal
>> >>> (752/800)
>> >>>
>> >>> ^^^^^^^ this is where it stays forever
>> >>>
>> >>> Looking at the Spark UI, several of the executors still list active
>> >>> tasks.  I do see that the Shuffle Read for executors that don't have
>> >>> any
>> >>> tasks remaining is around 100MB, whereas it's more like 10MB for the
>> >>> executors that still have tasks.
>> >>>
>> >>> The first stage, mapPartitions, always completes fine.  It's the
>> >>> second
>> >>> stage (takeOrdered), that hangs.
>> >>>
>> >>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
>> >>> encountered it when using JSON files (instead of Parquet).
>> >>>
>> >>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
>> >>> are having this issue.
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Christian Perez
>> Silicon Valley Data Science
>> Data Analyst
>> christian@svds.com
>> @cp_phd
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>



-- 
Christian Perez
Silicon Valley Data Science
Data Analyst
christian@svds.com
@cp_phd

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark SQL queries hang forever

Posted by Zsolt Tóth <to...@gmail.com>.
I have a very similar problem with collect() but only on relatively large
input data, on Amazon EMR. Look for my thread for details if interested.
I get "connection timed out" exception after a 15-20 minutes in one of the
container logs - does it happen in your case?

2015-03-30 23:34 GMT+02:00 Christian Perez <ch...@svds.com>:

> I'm having the exact same problem (hanging) with takeOrdered (using
> PySpark's orderBy on a DataFrame.)
>
> On Thu, Mar 26, 2015 at 7:21 PM, Jon Chase <jo...@gmail.com> wrote:
> > Here are the jstacks from a 3 worker cluster.  Two of the workers are
> > sitting with 4 active tasks, the other has no active tasks.  Not sure if
> it
> > makes a difference, but I only see an ExecutorLauncher instance via JPS
> on
> > the worker that has no active tasks.
> >
> > Here's the first stack dump, this machine has 4 active tasks:
> >
> > 2015-03-27 02:11:46
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> mode):
> >
> > "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
> > nid=0x43a3 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
> os_prio=0
> > tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
> [0x00007f91bad9a000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000589884fc0> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
> > nid=0x42e2 runnable [0x00007f91b4c54000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000005898aa000> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
> > nid=0x42e1 runnable [0x00007f91b5d56000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058ad5b5f8> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
> > nid=0x42e0 runnable [0x00007f91b5e57000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000005898aa0c0> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
> > nid=0x42df runnable [0x00007f91b5f58000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058ad5b6b8> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> > tid=0x00007f91d0773800 nid=0x427b waiting on condition
> [0x00007f91b6258000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000006cfda6800> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> > at
> >
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> > at
> >
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
> > tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition
> [0x00007f91b6b5a000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
> > waiting on condition [0x00007f91b6c5b000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x000000058af8e0b8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
> > runnable [0x00007f91baf9c000]
> >    java.lang.Thread.State: RUNNABLE
> > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> > at
> >
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> > at
> >
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0
> tid=0x00007f91d8118800
> > nid=0x41ee waiting on condition [0x00007f91bae9b000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> > at
> >
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
> > nid=0x41ed runnable [0x00007f91baa97000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x0000000589224af0> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
> > tid=0x00007f91d0019000 nid=0x41ea waiting on condition
> [0x00007f91bac98000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000006c8837750> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> > at
> >
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> > at
> >
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
> > tid=0x00007f91d0016800 nid=0x41e9 waiting on condition
> [0x00007f91ba16d000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000006cc53bff0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> > at
> >
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> > at
> >
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
> > tid=0x00007f91d0015000 nid=0x41e8 waiting on condition
> [0x00007f91b9f6b000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000006cfda69b0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> > at
> >
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> > at
> >
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
> > nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
> >
> > "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
> > nid=0x41c6 runnable [0x00007f91bbac7000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058984ee60> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
> > tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition
> [0x00007f91bbbc8000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000005898f7290> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
> > tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
> > at java.lang.Object.wait(Object.java:502)
> > at java.util.TimerThread.mainLoop(Timer.java:526)
> > - locked <0x000000058974cb00> (a java.util.TaskQueue)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
> > tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <0x0000000589904038> (a java.util.TaskQueue)
> > at java.lang.Object.wait(Object.java:502)
> > at java.util.TimerThread.mainLoop(Timer.java:526)
> > - locked <0x0000000589904038> (a java.util.TaskQueue)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
> tid=0x00007f91ed92d800
> > nid=0x41bf waiting on condition [0x00007f91bab98000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000589884e80> (a
> > scala.concurrent.forkjoin.ForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0
> tid=0x00007f91ed92c800
> > nid=0x41be waiting on condition [0x00007f91ba370000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> > at
> >
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O server boss #12" #39 daemon prio=5 os_prio=0
> tid=0x0000000000a02000
> > nid=0x41bd runnable [0x00007f91b9564000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
> > - locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
> > nid=0x41bc runnable [0x00007f91b9665000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
> > - locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
> > nid=0x41bb runnable [0x00007f91b9766000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
> > - locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
> > nid=0x41ba runnable [0x00007f91b9867000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
> > - locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
> > nid=0x41b9 runnable [0x00007f91b9968000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
> > - locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
> > nid=0x41b8 runnable [0x00007f91b9c69000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
> > - locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5
> > os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
> > [0x00007f91b9d6a000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000589884f20> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5
> > os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
> > [0x00007f91b9e6b000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000589884f20> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> > tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> > at
> >
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> > at
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
> > nid=0x419e runnable [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
> > nid=0x419d waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
> > nid=0x419c waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
> > nid=0x419b waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
> > nid=0x419a runnable [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> > tid=0x00007f91ec101000 nid=0x4199 waiting on condition
> [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198
> in
> > Object.wait() [0x00007f91d4254000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> > - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> >
> > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
> > nid=0x4197 in Object.wait() [0x00007f91d4355000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:502)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> > - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
> >
> > "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
> > condition [0x00007f91f1fdf000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000589a1b5f0> (a
> > java.util.concurrent.CountDownLatch$Sync)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> > at
> >
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> > at
> >
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> > at
> >
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> > at scala.concurrent.Await$.ready(package.scala:86)
> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> > at
> >
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> > at
> >
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> > at
> >
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> >
> > "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
> >
> > "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
> > nid=0x4191 runnable
> >
> > "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
> > nid=0x4192 runnable
> >
> > "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
> > nid=0x4193 runnable
> >
> > "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
> > nid=0x4194 runnable
> >
> > "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
> > nid=0x4195 runnable
> >
> > "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
> > waiting on condition
> >
> > JNI global references: 319
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Here's the next worker, also for active tasks:
> >
> > 2015-03-27 02:12:54
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> mode):
> >
> > "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
> > nid=0x44f1 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
> os_prio=0
> > tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
> [0x00007fcbbf87f000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000589510428> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
> > tid=0x00007fcbe4028800 nid=0x445f waiting on condition
> [0x00007fcbbf97f000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000006126cd358> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> > at
> >
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> > at
> >
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
> > nid=0x4459 runnable [0x00007fcbc88c7000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058a6b8c98> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
> > nid=0x4458 runnable [0x00007fcbc7897000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058a6e5a18> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
> > nid=0x4457 runnable [0x00007fcbc7998000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058aade110> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
> > nid=0x4456 runnable [0x00007fcbc7b9a000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058aade1d0> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
> > tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition
> [0x00007fcbc4886000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000615539dd8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> > at
> >
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> > at
> >
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
> > tid=0x00007fcbe4165000 nid=0x4453 waiting on condition
> [0x00007fcbca620000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000617bd4238> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> > at
> >
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> > at
> >
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> > tid=0x00007fcbe4164000 nid=0x4452 waiting on condition
> [0x00007fcbc87c5000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000006126cd538> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> > at
> >
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> > at
> >
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> > at
> >
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> > at
> >
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
> > tid=0x00007fcbdcff1800 nid=0x436e waiting on condition
> [0x00007fcbc4485000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
> > waiting on condition [0x00007fcbc4586000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x000000058adabe70> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
> > runnable [0x00007fcbc7e9d000]
> >    java.lang.Thread.State: RUNNABLE
> > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> > at
> >
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> > at
> >
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> tid=0x00007fcbf9608800
> > nid=0x4365 waiting on condition [0x00007fcbc89c8000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> > at
> >
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
> > nid=0x4364 runnable [0x00007fcbc8ac9000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x0000000589116ae0> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
> > nid=0x4341 waiting on condition [0x00007fcbc95f4000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
> >
> > "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
> > nid=0x4340 runnable [0x00007fcbc94f3000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058964c068> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> > tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition
> [0x00007fcbc97f6000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x000000058a6fa058> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> > tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <0x000000058928c438> (a java.util.TaskQueue)
> > at java.lang.Object.wait(Object.java:502)
> > at java.util.TimerThread.mainLoop(Timer.java:526)
> > - locked <0x000000058928c438> (a java.util.TaskQueue)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> > tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
> > at java.lang.Object.wait(Object.java:502)
> > at java.util.TimerThread.mainLoop(Timer.java:526)
> > - locked <0x000000058928c5a0> (a java.util.TaskQueue)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
> tid=0x00007fcbdc29d000
> > nid=0x433a waiting on condition [0x00007fcbc85c4000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> > at
> >
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> tid=0x00007fcbe4474800
> > nid=0x4338 waiting on condition [0x00007fcbc86c5000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x000000058a70a468> (a
> > scala.concurrent.forkjoin.ForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> tid=0x00007fcbdc29a000
> > nid=0x4337 runnable [0x00007fcbc6e8f000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
> > - locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
> > nid=0x4336 runnable [0x00007fcbc6f90000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
> > - locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
> > nid=0x4335 runnable [0x00007fcbc7091000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
> > - locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
> > nid=0x4334 runnable [0x00007fcbc7392000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
> > - locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
> > nid=0x4333 runnable [0x00007fcbc7493000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
> > - locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
> > nid=0x4332 runnable [0x00007fcbc7594000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
> > - locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> > os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
> > [0x00007fcbc7695000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000005893c1820> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> > os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
> > [0x00007fcbc7796000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000005893c1820> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> > tid=0x00007fcbf995d800 nid=0x432a waiting on condition
> [0x00007fcbc7d9c000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> > at
> >
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> > at
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
> > nid=0x4318 runnable [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
> > nid=0x4317 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
> > nid=0x4316 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
> > nid=0x4315 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
> > nid=0x4314 runnable [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> > tid=0x00007fcbf8103800 nid=0x4313 waiting on condition
> [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312
> in
> > Object.wait() [0x00007fcbe1be7000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> > - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> >
> > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
> > nid=0x4311 in Object.wait() [0x00007fcbe8150000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:502)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> > - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
> >
> > "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
> > condition [0x00007fcbff9b9000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000589585398> (a
> > java.util.concurrent.CountDownLatch$Sync)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> > at
> >
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> > at
> >
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> > at
> >
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> > at scala.concurrent.Await$.ready(package.scala:86)
> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> > at
> >
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> > at
> >
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> > at
> >
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> >
> > "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
> >
> > "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
> > nid=0x430b runnable
> >
> > "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
> > nid=0x430c runnable
> >
> > "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
> > nid=0x430d runnable
> >
> > "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
> > nid=0x430e runnable
> >
> > "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
> > nid=0x430f runnable
> >
> > "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
> > waiting on condition
> >
> > JNI global references: 375
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > And the final worker, no active tasks:
> >
> >
> > 2015-03-27 02:13:26
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> mode):
> >
> > "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
> > nid=0x3c5b waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
> os_prio=0
> > tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
> [0x00007f36d2e33000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000005895676e8> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
> > nid=0x3b88 runnable [0x00007f36dafb9000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058a86a7d8> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
> > nid=0x3b87 runnable [0x00007f36d7138000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058abea6c8> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
> > nid=0x3b86 runnable [0x00007f36daeb8000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058a86a898> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
> > nid=0x3b85 runnable [0x00007f36d7e3b000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058abea788> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
> > tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition
> [0x00007f36d7a39000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
> > waiting on condition [0x00007f36d7b3a000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x000000058aecd008> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
> > runnable [0x00007f36db4be000]
> >    java.lang.Thread.State: RUNNABLE
> > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> > at
> >
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> > at
> >
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> tid=0x00007f36f8025000
> > nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> > at
> >
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
> > nid=0x3a96 runnable [0x00007f36dc0ea000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x0000000589202c58> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
> > nid=0x3a6d waiting on condition [0x00007f36dcc15000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
> >
> > "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
> > nid=0x3a6c runnable [0x00007f36dcb14000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058a825c18> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> > tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition
> [0x00007f36dce17000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000005897334d0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> > tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
> > at java.lang.Object.wait(Object.java:502)
> > at java.util.TimerThread.mainLoop(Timer.java:526)
> > - locked <0x00000005896b0d20> (a java.util.TaskQueue)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> > tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
> > at java.lang.Object.wait(Object.java:502)
> > at java.util.TimerThread.mainLoop(Timer.java:526)
> > - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
> tid=0x00007f36f03a1000
> > nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> > at
> >
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> tid=0x00007f370d9b2000
> > nid=0x3a64 waiting on condition [0x00007f36dbde7000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000005895675a8> (a
> > scala.concurrent.forkjoin.ForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> tid=0x00007f36f8713000
> > nid=0x3a63 runnable [0x00007f36da4b0000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
> > - locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
> > nid=0x3a62 runnable [0x00007f36da5b1000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
> > - locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
> > nid=0x3a61 runnable [0x00007f36da6b2000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
> > - locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
> > nid=0x3a60 runnable [0x00007f36da7b3000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
> > - locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
> > nid=0x3a5f runnable [0x00007f36da8b4000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
> > - locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
> > nid=0x3a5e runnable [0x00007f36da9b5000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
> > - locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> > os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
> > [0x00007f36dacb6000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000589567648> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> > os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
> > [0x00007f36dadb7000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x0000000589567648> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> > tid=0x00007f370d97c000 nid=0x3a56 waiting on condition
> [0x00007f36db3bd000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> > at
> >
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> > at
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
> > nid=0x3a43 runnable [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
> > nid=0x3a42 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
> > nid=0x3a41 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
> > nid=0x3a40 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
> > nid=0x3a3f runnable [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> > tid=0x00007f370c101800 nid=0x3a3e waiting on condition
> [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d
> in
> > Object.wait() [0x00007f36f51e5000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> > - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> >
> > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
> > nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:502)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> > - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
> >
> > "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
> > condition [0x00007f3712fda000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x000000058a974ab8> (a
> > java.util.concurrent.CountDownLatch$Sync)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> > at
> >
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> > at
> >
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> > at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> > at
> >
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> > at scala.concurrent.Await$.ready(package.scala:86)
> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> > at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> > at
> >
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> > at
> >
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> > at
> >
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> > at
> >
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> >
> > "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
> >
> > "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
> > nid=0x3a34 runnable
> >
> > "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
> > nid=0x3a35 runnable
> >
> > "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
> > nid=0x3a36 runnable
> >
> > "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
> > nid=0x3a37 runnable
> >
> > "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
> > nid=0x3a3a runnable
> >
> > "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
> > waiting on condition
> >
> > JNI global references: 486
> >
> >
> >
> >
> >
> >
> >
> >
> > And the driver:
> >
> > 2015-03-27 02:20:47
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> mode):
> >
> > "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
> > nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e67194f0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
> > nid=0x465a waiting on condition [0x00007fd3b8eb5000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e67194f0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
> > nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e67194f0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
> > nid=0x41d0 waiting on condition [0x00007fd38feef000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e67194f0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
> > tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition
> [0x00007fd38fff0000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000fd545eb8> (a
> > scala.concurrent.forkjoin.ForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5
> os_prio=0
> > tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition
> [0x00007fd3b8400000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e64f49f8> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
> > nid=0x417f waiting on condition [0x00007fd398935000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e67194f0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "task-result-getter-3" #107 daemon prio=5 os_prio=0
> tid=0x00007fd3b003f800
> > nid=0x409c waiting on condition [0x00007fd3902f1000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e66e9a00> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "task-result-getter-2" #106 daemon prio=5 os_prio=0
> tid=0x00007fd3b003d000
> > nid=0x409b waiting on condition [0x00007fd3903f2000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e66e9a00> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "task-result-getter-1" #105 daemon prio=5 os_prio=0
> tid=0x00007fd3b003b800
> > nid=0x409a waiting on condition [0x00007fd3904f3000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e66e9a00> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "task-result-getter-0" #104 daemon prio=5 os_prio=0
> tid=0x00007fd3b0030800
> > nid=0x4098 waiting on condition [0x00007fd3905f4000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e66e9a00> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
> tid=0x00007fd3cc00c800
> > nid=0x4093 waiting on condition [0x00007fd3936f8000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> > at
> >
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
> > nid=0x4092 runnable [0x00007fd398b37000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e685f320> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
> > nid=0x4091 runnable [0x00007fd398a36000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e6871c30> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
> > nid=0x4090 runnable [0x00007fd39903a000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e6873520> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
> > tid=0x00007fd3e0011800 nid=0x4056 waiting on condition
> [0x00007fd398733000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
> > tid=0x00007fd3dc001800 nid=0x4055 waiting on condition
> [0x00007fd398834000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e7f19070> (a
> > scala.concurrent.forkjoin.ForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5
> os_prio=0
> > tid=0x00007fd3a400d800 nid=0x4051 waiting on condition
> [0x00007fd398e38000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e64f49f8> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
> > tid=0x00007fd432c05000 nid=0x404f waiting on condition
> [0x00007fd398f39000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e66fe220> (a
> > java.util.concurrent.SynchronousQueue$TransferStack)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> > at
> >
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
> > tid=0x00007fd43297c800 nid=0x4042 waiting on condition
> [0x00007fd39973b000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e740fa58> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> > at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
> > tid=0x00007fd4329ac000 nid=0x4041 waiting on condition
> [0x00007fd39983c000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e740fce8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
> > tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> > - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> > at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
> > tid=0x00007fd4328e6800 nid=0x403b waiting on condition
> [0x00007fd399c3e000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e737a0c8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> > at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
> > tid=0x00007fd4328eb800 nid=0x403a waiting on condition
> [0x00007fd399d3f000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e737add8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
> > tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> > - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> > at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
> > tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
> >    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> > - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
> > at
> >
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
> >
> > "ResponseProcessor for block
> > BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
> > prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
> > [0x00007fd39a941000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
> > - locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
> > at
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> > at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> > at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> > at java.io.FilterInputStream.read(FilterInputStream.java:83)
> > at java.io.FilterInputStream.read(FilterInputStream.java:83)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
> > at
> >
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
> >
> > "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
> > nid=0x3fda waiting on condition [0x00007fd39aa42000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e58145f0> (a
> > java.util.concurrent.Semaphore$NonfairSync)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> > at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> > at
> >
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
> > at
> >
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> > at
> >
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> > at
> >
> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
> >
> > "Spark Context Cleaner" #85 daemon prio=5 os_prio=0
> tid=0x00007fd43258c000
> > nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
> >    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> > - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
> > at
> >
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
> > at
> >
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> > at
> >
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> > at
> > org.apache.spark.ContextCleaner.org
> $apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
> > at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
> >
> > "DataStreamer for file
> /spark-logs/application_1427416187821_0004.inprogress
> > block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84
> > daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in
> Object.wait()
> > [0x00007fd39ae44000]
> >    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
> > - locked <0x00000000e6994f30> (a java.util.LinkedList)
> >
> > "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
> > tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition
> [0x00007fd39af45000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e6995e88> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
> > nid=0x3fb7 runnable [0x00007fd39b246000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e6886b58> (a
> > io.netty.channel.nio.SelectedSelectionKeySet)
> > - locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Yarn application state monitor" #73 daemon prio=5 os_prio=0
> > tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition
> [0x00007fd39b8fb000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
> >
> > "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
> > nid=0x3f77 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f
> > waiting on condition [0x00007fd39b547000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e677ae98> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
> > tid=0x00007fd432896800 nid=0x3e6a waiting on condition
> [0x00007fd39b9fc000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
> > at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> > at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e
> > runnable [0x00007fd3b80ff000]
> >    java.lang.Thread.State: RUNNABLE
> > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> > at
> >
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> > at
> >
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop"
> #62
> > daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in
> Object.wait()
> > [0x00007fd3b8501000]
> >    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
> > - locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
> >
> > "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
> > tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition
> [0x00007fd3b8602000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e66e9230> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> > at
> >
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> > at
> >
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
> >
> > "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f
> in
> > Object.wait() [0x00007fd3b8703000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:502)
> > at java.util.TimerThread.mainLoop(Timer.java:526)
> > - locked <0x00000000e66e8190> (a java.util.TaskQueue)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
> > nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e67194f0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54
> daemon
> > prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
> > [0x00007fd3b8fb6000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> > at
> >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> > at
> >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> > - locked <0x00000000e671a620> (a java.lang.Object)
> > at
> >
> org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> > at
> >
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
> > tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
> > - locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> > at
> >
> org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
> > tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:502)
> > at java.util.TimerThread.mainLoop(Timer.java:526)
> > - locked <0x00000000e66609f0> (a java.util.TaskQueue)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
> > nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e6539dc8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
> > nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e6539dc8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
> > nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e6539dc8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
> > nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e6539dc8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
> > nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e6539dc8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
> > nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e6539dc8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
> > nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e6539dc8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon
> prio=5
> > os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable [0x00007fd3ba0de000]
> >    java.lang.Thread.State: RUNNABLE
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > at java.net.ServerSocket.accept(ServerSocket.java:513)
> > at
> >
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> > at
> >
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
> > tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:502)
> > at java.util.TimerThread.mainLoop(Timer.java:526)
> > - locked <0x00000000e6601270> (a java.util.TaskQueue)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
> > tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:502)
> > at java.util.TimerThread.mainLoop(Timer.java:526)
> > - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
> > tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:502)
> > at java.util.TimerThread.mainLoop(Timer.java:526)
> > - locked <0x00000000e65380d8> (a java.util.TaskQueue)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > "New I/O server boss #6" #36 daemon prio=5 os_prio=0
> tid=0x00007fd39c04b800
> > nid=0x3df8 runnable [0x00007fd3ba6e2000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
> > - locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
> > nid=0x3df7 runnable [0x00007fd3ba7e3000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
> > - locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
> > nid=0x3df6 runnable [0x00007fd3ba8e4000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
> > - locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
> > nid=0x3df5 runnable [0x00007fd3ba9e5000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
> > - locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
> > nid=0x3df4 runnable [0x00007fd3baae6000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
> > - locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
> > nid=0x3df3 runnable [0x00007fd3babe7000]
> >    java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
> > - locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
> > - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > at
> >
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> > at
> >
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> >
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> >
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
> > os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
> > [0x00007fd3baef9000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e64bbb40> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
> > os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
> > [0x00007fd3baffa000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e64bbb40> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0
> > tid=0x00007fd431f99000 nid=0x3de0 waiting on condition
> [0x00007fd3bb0fb000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e64f49f8> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0
> > tid=0x00007fd431f96800 nid=0x3ddf waiting on condition
> [0x00007fd3bb1fc000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e64f49f8> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0
> > tid=0x00007fd431f86000 nid=0x3dde waiting on condition
> [0x00007fd3bb2fd000]
> >    java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e64f49f8> (a
> > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
> tid=0x00007fd431f0e800
> > nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> > at
> >
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> > at
> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
> > nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e42808f8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
> > nid=0x3d75 waiting on condition [0x00007fd40c15d000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e42808f8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
> > nid=0x3d74 waiting on condition [0x00007fd40c25e000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e42808f8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
> > nid=0x3d73 waiting on condition [0x00007fd40c35f000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e42808f8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
> > nid=0x3d72 waiting on condition [0x00007fd40c460000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e42808f8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
> > nid=0x3d71 waiting on condition [0x00007fd40c561000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e42808f8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
> > nid=0x3d70 waiting on condition [0x00007fd40c662000]
> >    java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000e42808f8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> > at
> >
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon
> prio=5
> > os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable [0x00007fd40c966000]
> >    java.lang.Thread.State: RUNNABLE
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > at java.net.ServerSocket.accept(ServerSocket.java:513)
> > at
> >
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> > at
> >
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> > at
> >
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
> > nid=0x3d67 runnable [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
> > nid=0x3d66 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
> > nid=0x3d65 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
> > nid=0x3d64 waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
> > nid=0x3d63 runnable [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62
> in
> > Object.wait() [0x00007fd40f28e000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> > - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> >
> > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
> > nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:502)
> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> > - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
> >
> > "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
> > Object.wait() [0x00007fd438220000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:502)
> > at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> > - locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
> > at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
> > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
> > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
> > at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
> > at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
> > at
> >
> org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
> > at
> >
> org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
> > at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
> > at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
> > at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
> > at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
> > at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
> > at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
> > at
> >
> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
> > at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
> > at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
> > at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
> > at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
> > at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
> > at $line24.$read$$iwC$$iwC.<init>(<console>:35)
> > at $line24.$read$$iwC.<init>(<console>:37)
> > at $line24.$read.<init>(<console>:39)
> > at $line24.$read$.<init>(<console>:43)
> > at $line24.$read$.<clinit>(<console>)
> > at $line24.$eval$.<init>(<console>:7)
> > at $line24.$eval$.<clinit>(<console>)
> > at $line24.$eval.$print(<console>)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > at
> >
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> > at
> >
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
> > at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
> > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> > at
> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
> > at
> >
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
> > at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
> > at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
> > at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
> > at
> > org.apache.spark.repl.SparkILoop.org
> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
> > at
> >
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
> > at
> >
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> > at
> >
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> > at
> >
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> > at
> > org.apache.spark.repl.SparkILoop.org
> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
> > at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
> > at org.apache.spark.repl.Main$.main(Main.scala:31)
> > at org.apache.spark.repl.Main.main(Main.scala)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > at
> >
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
> > at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
> > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
> > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
> > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> >
> > "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
> >
> > "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000
> nid=0x3d5c
> > runnable
> >
> > "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800
> nid=0x3d5d
> > runnable
> >
> > "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800
> nid=0x3d5e
> > runnable
> >
> > "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000
> nid=0x3d5f
> > runnable
> >
> > "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
> > waiting on condition
> >
> > JNI global references: 420
> >
> >
> >
> > On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <
> michael@databricks.com>
> > wrote:
> >>
> >> Is it possible to jstack the executors and see where they are hanging?
> >>
> >> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:
> >>>
> >>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu,
> 30GB),
> >>> executor memory 20GB, driver memory 10GB
> >>>
> >>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data
> spread
> >>> out over roughly 2,000 Parquet files and my queries frequently hang.
> Simple
> >>> queries like "select count(*) from ..." on the entire data set work ok.
> >>> Slightly more demanding ones with group by's and some aggregate
> functions
> >>> (percentile_approx, avg, etc.) work ok as well, as long as I have some
> >>> criteria in my where clause to keep the number of rows down.
> >>>
> >>> Once I hit some limit on query complexity and rows processed, my
> queries
> >>> start to hang.  I've left them for up to an hour without seeing any
> >>> progress.  No OOM's either - the job is just stuck.
> >>>
> >>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800,
> but
> >>> with the same results: usually near the end of the tasks (like 780 of
> 800
> >>> complete), progress just stops:
> >>>
> >>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in
> >>> stage 1.0 (TID 1618) in 800 ms on
> >>> ip-10-209-22-211.eu-west-1.compute.internal (748/800)
> >>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in
> >>> stage 1.0 (TID 1623) in 622 ms on
> ip-10-105-12-41.eu-west-1.compute.internal
> >>> (749/800)
> >>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in
> >>> stage 1.0 (TID 1627) in 616 ms on
> ip-10-90-2-201.eu-west-1.compute.internal
> >>> (750/800)
> >>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in
> >>> stage 1.0 (TID 1629) in 611 ms on
> ip-10-90-2-201.eu-west-1.compute.internal
> >>> (751/800)
> >>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in
> >>> stage 1.0 (TID 1625) in 669 ms on
> ip-10-105-12-41.eu-west-1.compute.internal
> >>> (752/800)
> >>>
> >>> ^^^^^^^ this is where it stays forever
> >>>
> >>> Looking at the Spark UI, several of the executors still list active
> >>> tasks.  I do see that the Shuffle Read for executors that don't have
> any
> >>> tasks remaining is around 100MB, whereas it's more like 10MB for the
> >>> executors that still have tasks.
> >>>
> >>> The first stage, mapPartitions, always completes fine.  It's the second
> >>> stage (takeOrdered), that hangs.
> >>>
> >>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
> >>> encountered it when using JSON files (instead of Parquet).
> >>>
> >>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
> >>> are having this issue.
> >>
> >>
> >
>
>
>
> --
> Christian Perez
> Silicon Valley Data Science
> Data Analyst
> christian@svds.com
> @cp_phd
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: Spark SQL queries hang forever

Posted by Christian Perez <ch...@svds.com>.
I'm having the exact same problem (hanging) with takeOrdered (using
PySpark's orderBy on a DataFrame.)

On Thu, Mar 26, 2015 at 7:21 PM, Jon Chase <jo...@gmail.com> wrote:
> Here are the jstacks from a 3 worker cluster.  Two of the workers are
> sitting with 4 active tasks, the other has no active tasks.  Not sure if it
> makes a difference, but I only see an ExecutorLauncher instance via JPS on
> the worker that has no active tasks.
>
> Here's the first stack dump, this machine has 4 active tasks:
>
> 2015-03-27 02:11:46
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
> nid=0x43a3 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5 os_prio=0
> tid=0x00007f91d02fd000 nid=0x42ee waiting on condition [0x00007f91bad9a000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884fc0> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
> nid=0x42e2 runnable [0x00007f91b4c54000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005898aa000> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
> nid=0x42e1 runnable [0x00007f91b5d56000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058ad5b5f8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
> nid=0x42e0 runnable [0x00007f91b5e57000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005898aa0c0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
> nid=0x42df runnable [0x00007f91b5f58000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058ad5b6b8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cfda6800> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
> tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
> waiting on condition [0x00007f91b6c5b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058af8e0b8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
> runnable [0x00007f91baf9c000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0 tid=0x00007f91d8118800
> nid=0x41ee waiting on condition [0x00007f91bae9b000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
> nid=0x41ed runnable [0x00007f91baa97000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589224af0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
> tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006c8837750> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
> tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cc53bff0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
> tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cfda69b0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
> nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
> nid=0x41c6 runnable [0x00007f91bbac7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058984ee60> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
> tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005898f7290> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
> tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058974cb00> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
> tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000589904038> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x0000000589904038> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0 tid=0x00007f91ed92d800
> nid=0x41bf waiting on condition [0x00007f91bab98000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884e80> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800
> nid=0x41be waiting on condition [0x00007f91ba370000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O server boss #12" #39 daemon prio=5 os_prio=0 tid=0x0000000000a02000
> nid=0x41bd runnable [0x00007f91b9564000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
> - locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
> nid=0x41bc runnable [0x00007f91b9665000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
> - locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
> nid=0x41bb runnable [0x00007f91b9766000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
> - locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
> nid=0x41ba runnable [0x00007f91b9867000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
> nid=0x41b9 runnable [0x00007f91b9968000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
> - locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
> nid=0x41b8 runnable [0x00007f91b9c69000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
> - locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5
> os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
> [0x00007f91b9d6a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884f20> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5
> os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
> [0x00007f91b9e6b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884f20> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
> nid=0x419e runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
> nid=0x419d waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
> nid=0x419c waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
> nid=0x419b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
> nid=0x419a runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198 in
> Object.wait() [0x00007f91d4254000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
> nid=0x4197 in Object.wait() [0x00007f91d4355000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
> condition [0x00007f91f1fdf000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589a1b5f0> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
> nid=0x4191 runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
> nid=0x4192 runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
> nid=0x4193 runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
> nid=0x4194 runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
> nid=0x4195 runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
> waiting on condition
>
> JNI global references: 319
>
>
>
>
>
>
>
>
>
>
>
>
> Here's the next worker, also for active tasks:
>
> 2015-03-27 02:12:54
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
> nid=0x44f1 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5 os_prio=0
> tid=0x00007fcbe488b000 nid=0x4464 waiting on condition [0x00007fcbbf87f000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589510428> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
> tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006126cd358> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
> nid=0x4459 runnable [0x00007fcbc88c7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a6b8c98> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
> nid=0x4458 runnable [0x00007fcbc7897000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a6e5a18> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
> nid=0x4457 runnable [0x00007fcbc7998000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058aade110> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
> nid=0x4456 runnable [0x00007fcbc7b9a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058aade1d0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
> tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000615539dd8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
> tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000617bd4238> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006126cd538> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
> tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
> waiting on condition [0x00007fcbc4586000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058adabe70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
> runnable [0x00007fcbc7e9d000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007fcbf9608800
> nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
> nid=0x4364 runnable [0x00007fcbc8ac9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589116ae0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
> nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
> nid=0x4340 runnable [0x00007fcbc94f3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058964c068> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a6fa058> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058928c438> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058928c438> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058928c5a0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000
> nid=0x433a waiting on condition [0x00007fcbc85c4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007fcbe4474800
> nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a70a468> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007fcbdc29a000
> nid=0x4337 runnable [0x00007fcbc6e8f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
> - locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
> nid=0x4336 runnable [0x00007fcbc6f90000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
> - locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
> nid=0x4335 runnable [0x00007fcbc7091000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
> - locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
> nid=0x4334 runnable [0x00007fcbc7392000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
> - locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
> nid=0x4333 runnable [0x00007fcbc7493000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
> - locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
> nid=0x4332 runnable [0x00007fcbc7594000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
> - locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
> [0x00007fcbc7695000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005893c1820> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
> [0x00007fcbc7796000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005893c1820> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
> nid=0x4318 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
> nid=0x4317 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
> nid=0x4316 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
> nid=0x4315 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
> nid=0x4314 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312 in
> Object.wait() [0x00007fcbe1be7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
> nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
> condition [0x00007fcbff9b9000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589585398> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
> nid=0x430b runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
> nid=0x430c runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
> nid=0x430d runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
> nid=0x430e runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
> nid=0x430f runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
> waiting on condition
>
> JNI global references: 375
>
>
>
>
>
>
>
>
>
>
> And the final worker, no active tasks:
>
>
> 2015-03-27 02:13:26
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
> nid=0x3c5b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5 os_prio=0
> tid=0x00007f36f072d000 nid=0x3b9a waiting on condition [0x00007f36d2e33000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005895676e8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
> nid=0x3b88 runnable [0x00007f36dafb9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a86a7d8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
> nid=0x3b87 runnable [0x00007f36d7138000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058abea6c8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
> nid=0x3b86 runnable [0x00007f36daeb8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a86a898> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
> nid=0x3b85 runnable [0x00007f36d7e3b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058abea788> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
> tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
> waiting on condition [0x00007f36d7b3a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058aecd008> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
> runnable [0x00007f36db4be000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007f36f8025000
> nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
> nid=0x3a96 runnable [0x00007f36dc0ea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589202c58> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
> nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
> nid=0x3a6c runnable [0x00007f36dcb14000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a825c18> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005897334d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000005896b0d20> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000
> nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007f370d9b2000
> nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005895675a8> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007f36f8713000
> nid=0x3a63 runnable [0x00007f36da4b0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
> - locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
> nid=0x3a62 runnable [0x00007f36da5b1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
> - locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
> nid=0x3a61 runnable [0x00007f36da6b2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
> - locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
> nid=0x3a60 runnable [0x00007f36da7b3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
> - locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
> nid=0x3a5f runnable [0x00007f36da8b4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
> - locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
> nid=0x3a5e runnable [0x00007f36da9b5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
> - locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
> [0x00007f36dacb6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589567648> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
> [0x00007f36dadb7000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589567648> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
> nid=0x3a43 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
> nid=0x3a42 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
> nid=0x3a41 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
> nid=0x3a40 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
> nid=0x3a3f runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d in
> Object.wait() [0x00007f36f51e5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
> nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
> condition [0x00007f3712fda000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a974ab8> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
> nid=0x3a34 runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
> nid=0x3a35 runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
> nid=0x3a36 runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
> nid=0x3a37 runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
> nid=0x3a3a runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
> waiting on condition
>
> JNI global references: 486
>
>
>
>
>
>
>
>
> And the driver:
>
> 2015-03-27 02:20:47
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
> nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
> nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
> nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
> nid=0x41d0 waiting on condition [0x00007fd38feef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
> tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000fd545eb8> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5 os_prio=0
> tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition [0x00007fd3b8400000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
> nid=0x417f waiting on condition [0x00007fd398935000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800
> nid=0x409c waiting on condition [0x00007fd3902f1000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000
> nid=0x409b waiting on condition [0x00007fd3903f2000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800
> nid=0x409a waiting on condition [0x00007fd3904f3000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800
> nid=0x4098 waiting on condition [0x00007fd3905f4000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0 tid=0x00007fd3cc00c800
> nid=0x4093 waiting on condition [0x00007fd3936f8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
> nid=0x4092 runnable [0x00007fd398b37000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e685f320> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
> nid=0x4091 runnable [0x00007fd398a36000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6871c30> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
> nid=0x4090 runnable [0x00007fd39903a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6873520> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
> tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
> tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e7f19070> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0
> tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
> tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66fe220> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
> tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e740fa58> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
> tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e740fce8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
> tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
> tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e737a0c8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
> tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e737add8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
> tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> at java.lang.Thread.run(Thread.java:745)
>
> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
> tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
> at
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>
> "ResponseProcessor for block
> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
> prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
> [0x00007fd39a941000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
> - locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>
> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
> nid=0x3fda waiting on condition [0x00007fd39aa42000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e58145f0> (a
> java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>
> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000
> nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at
> org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
> at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>
> "DataStreamer for file /spark-logs/application_1427416187821_0004.inprogress
> block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84
> daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait()
> [0x00007fd39ae44000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
> - locked <0x00000000e6994f30> (a java.util.LinkedList)
>
> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
> tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6995e88> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
> nid=0x3fb7 runnable [0x00007fd39b246000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6886b58> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Yarn application state monitor" #73 daemon prio=5 os_prio=0
> tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>
> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
> nid=0x3f77 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f
> waiting on condition [0x00007fd39b547000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e677ae98> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
> tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e
> runnable [0x00007fd3b80ff000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop" #62
> daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in Object.wait()
> [0x00007fd3b8501000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
> - locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>
> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
> tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9230> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> at
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>
> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in
> Object.wait() [0x00007fd3b8703000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e66e8190> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
> nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54 daemon
> prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
> [0x00007fd3b8fb6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> - locked <0x00000000e671a620> (a java.lang.Object)
> at
> org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
> tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> at
> org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
> tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e66609f0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
> nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
> nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
> nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
> nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
> nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
> nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
> nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon prio=5
> os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable [0x00007fd3ba0de000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
> tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e6601270> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
> tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
> tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e65380d8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "New I/O server boss #6" #36 daemon prio=5 os_prio=0 tid=0x00007fd39c04b800
> nid=0x3df8 runnable [0x00007fd3ba6e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
> nid=0x3df7 runnable [0x00007fd3ba7e3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
> nid=0x3df6 runnable [0x00007fd3ba8e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
> nid=0x3df5 runnable [0x00007fd3ba9e5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
> - locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
> nid=0x3df4 runnable [0x00007fd3baae6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
> - locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
> nid=0x3df3 runnable [0x00007fd3babe7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
> - locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
> os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
> [0x00007fd3baef9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64bbb40> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
> os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
> [0x00007fd3baffa000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64bbb40> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0
> tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0
> tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0
> tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0 tid=0x00007fd431f0e800
> nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
> nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
> nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
> nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
> nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
> nid=0x3d72 waiting on condition [0x00007fd40c460000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
> nid=0x3d71 waiting on condition [0x00007fd40c561000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
> nid=0x3d70 waiting on condition [0x00007fd40c662000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon prio=5
> os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable [0x00007fd40c966000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
> nid=0x3d67 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
> nid=0x3d66 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
> nid=0x3d65 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
> nid=0x3d64 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
> nid=0x3d63 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62 in
> Object.wait() [0x00007fd40f28e000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
> nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
> Object.wait() [0x00007fd438220000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> - locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
> at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
> at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
> at
> org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
> at
> org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
> at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
> at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
> at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
> at
> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
> at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
> at $line24.$read$$iwC$$iwC.<init>(<console>:35)
> at $line24.$read$$iwC.<init>(<console>:37)
> at $line24.$read.<init>(<console>:39)
> at $line24.$read$.<init>(<console>:43)
> at $line24.$read$.<clinit>(<console>)
> at $line24.$eval$.<init>(<console>:7)
> at $line24.$eval$.<clinit>(<console>)
> at $line24.$eval.$print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
> at
> org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at
> org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000 nid=0x3d5c
> runnable
>
> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800 nid=0x3d5d
> runnable
>
> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800 nid=0x3d5e
> runnable
>
> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000 nid=0x3d5f
> runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
> waiting on condition
>
> JNI global references: 420
>
>
>
> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>>
>> Is it possible to jstack the executors and see where they are hanging?
>>
>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:
>>>
>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB),
>>> executor memory 20GB, driver memory 10GB
>>>
>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data spread
>>> out over roughly 2,000 Parquet files and my queries frequently hang. Simple
>>> queries like "select count(*) from ..." on the entire data set work ok.
>>> Slightly more demanding ones with group by's and some aggregate functions
>>> (percentile_approx, avg, etc.) work ok as well, as long as I have some
>>> criteria in my where clause to keep the number of rows down.
>>>
>>> Once I hit some limit on query complexity and rows processed, my queries
>>> start to hang.  I've left them for up to an hour without seeing any
>>> progress.  No OOM's either - the job is just stuck.
>>>
>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but
>>> with the same results: usually near the end of the tasks (like 780 of 800
>>> complete), progress just stops:
>>>
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in
>>> stage 1.0 (TID 1618) in 800 ms on
>>> ip-10-209-22-211.eu-west-1.compute.internal (748/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in
>>> stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>> (749/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in
>>> stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>> (750/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in
>>> stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>> (751/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in
>>> stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>> (752/800)
>>>
>>> ^^^^^^^ this is where it stays forever
>>>
>>> Looking at the Spark UI, several of the executors still list active
>>> tasks.  I do see that the Shuffle Read for executors that don't have any
>>> tasks remaining is around 100MB, whereas it's more like 10MB for the
>>> executors that still have tasks.
>>>
>>> The first stage, mapPartitions, always completes fine.  It's the second
>>> stage (takeOrdered), that hangs.
>>>
>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
>>> encountered it when using JSON files (instead of Parquet).
>>>
>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
>>> are having this issue.
>>
>>
>



-- 
Christian Perez
Silicon Valley Data Science
Data Analyst
christian@svds.com
@cp_phd

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark SQL queries hang forever

Posted by Michael Allman <mi...@videoamp.com>.
Hi Reynold,

Our build is based off of the 1.3 branch as of commit 6d3c4d8b04b2738a821dfcc3df55a5635b89e506. Unless your fix is on another branch it appears this problem still exists.

LMK if there's anything else I can help with.

Thanks,

Michael

> On Apr 16, 2015, at 11:44 AM, Reynold Xin <rx...@databricks.com> wrote:
> 
> Thanks for the information.
> 
> Adding the dev list.
> 
> Do you mind trying 1.3.1 to see if the issue is fixed?  I could be wrong but it migtht be related to https://issues.apache.org/jira/browse/SPARK-6578 <https://issues.apache.org/jira/browse/SPARK-6578>
> 
>  
> 
> On Thu, Apr 16, 2015 at 2:34 AM, Michael Allman <michael@videoamp.com <ma...@videoamp.com>> wrote:
> After many, many manual (and automated) trials using a spark sql query that *sometimes* triggers the hang I found the following:
> 
> When I set spark.shuffle.blockTransferService to netty (the default), I can sometimes reproduce the problem.
> When I set spark.shuffle.blockTransferService to nio (the Spark 1.1 default), I have not yet been able to reproduce the problem.
> 
> While far from a reliable conclusion, the workaround I'm going with for now is setting "spark.shuffle.blockTransferService nio" in spark-defaults.conf. One of the reasons I homed in on this setting as a possible trigger was the fact that its default value changed starting in Spark 1.2. We haven't experienced a similar problem in our current Spark 1.1 cluster, and we're skipping directly to Spark 1.3. Has anyone else tried this apparent workaround?
> 
> Cheers,
> 
> Michael
> 
>> On Apr 15, 2015, at 11:23 PM, Michael Allman <michael@videoamp.com <ma...@videoamp.com>> wrote:
>> 
>> FWIW I'm attempting an upgrade from Spark 1.1 to Spark 1.3 and I'm seeing this exact problem in ShuffleBlockFetcherIterator in the query takeOrdered stage. Sometimes a single task hangs, sometimes a couple dozen. The final stage has 200 tasks. I'm using the Spark standalone scheduler, parquet files. Here's the stack trace of the stuck thread:
>> 
>> sun.misc.Unsafe.park(Native Method)
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:389)
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:154)
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:149)
>> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> org.apache.spark.scheduler.Task.run(Task.scala:64)
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> java.lang.Thread.run(Thread.java:745)
>> 
>> Anyone find a reliable fix/workaround? I'm going to try some things myself. If I find something that works I'll post back.
>> 
>> Cheers,
>> 
>> Michael
>> 
>> 
>>> On Mar 26, 2015, at 7:21 PM, Jon Chase <jon.chase@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> Here are the jstacks from a 3 worker cluster.  Two of the workers are sitting with 4 active tasks, the other has no active tasks.  Not sure if it makes a difference, but I only see an ExecutorLauncher instance via JPS on the worker that has no active tasks.  
>>> 
>>> Here's the first stack dump, this machine has 4 active tasks:
>>> 
>>> 2015-03-27 02:11:46
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000 nid=0x43a3 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5 os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition [0x00007f91bad9a000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884fc0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000 nid=0x42e2 runnable [0x00007f91b4c54000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005898aa000> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800 nid=0x42e1 runnable [0x00007f91b5d56000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058ad5b5f8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000 nid=0x42e0 runnable [0x00007f91b5e57000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005898aa0c0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800 nid=0x42df runnable [0x00007f91b5f58000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058ad5b6b8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cfda6800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0 tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2 waiting on condition [0x00007f91b6c5b000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058af8e0b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef runnable [0x00007f91baf9c000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0 tid=0x00007f91d8118800 nid=0x41ee waiting on condition [0x00007f91bae9b000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000 nid=0x41ed runnable [0x00007f91baa97000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589224af0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0 tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006c8837750> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0 tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cc53bff0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0 tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cfda69b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800 nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800 nid=0x41c6 runnable [0x00007f91bbac7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058984ee60> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0 tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005898f7290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0 tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058974cb00> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058974cb00> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0 tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x0000000589904038> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x0000000589904038> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0 tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884e80> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800 nid=0x41be waiting on condition [0x00007f91ba370000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O server boss #12" #39 daemon prio=5 os_prio=0 tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058992f188> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000 nid=0x41bc runnable [0x00007f91b9665000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589931218> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800 nid=0x41bb runnable [0x00007f91b9766000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800 nid=0x41ba runnable [0x00007f91b9867000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800 nid=0x41b9 runnable [0x00007f91b9968000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000 nid=0x41b8 runnable [0x00007f91b9c69000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5 os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition [0x00007f91b9d6a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5 os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition [0x00007f91b9e6b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800 nid=0x419e runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000 nid=0x419d waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000 nid=0x419c waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000 nid=0x419b waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800 nid=0x419a runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198 in Object.wait() [0x00007f91d4254000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800 nid=0x4197 in Object.wait() [0x00007f91d4355000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on condition [0x00007f91f1fdf000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589a1b5f0> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000 nid=0x4191 runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800 nid=0x4192 runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800 nid=0x4193 runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000 nid=0x4194 runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000 nid=0x4195 runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f waiting on condition
>>> 
>>> JNI global references: 319
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Here's the next worker, also for active tasks:
>>> 
>>> 2015-03-27 02:12:54
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000 nid=0x44f1 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5 os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition [0x00007fcbbf87f000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589510428> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0 tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006126cd358> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800 nid=0x4459 runnable [0x00007fcbc88c7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a6b8c98> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000 nid=0x4458 runnable [0x00007fcbc7897000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a6e5a18> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800 nid=0x4457 runnable [0x00007fcbc7998000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058aade110> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000 nid=0x4456 runnable [0x00007fcbc7b9a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058aade1d0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0 tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000615539dd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0 tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000617bd4238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006126cd538> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0 tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369 waiting on condition [0x00007fcbc4586000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058adabe70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366 runnable [0x00007fcbc7e9d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007fcbf9608800 nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800 nid=0x4364 runnable [0x00007fcbc8ac9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589116ae0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000 nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000 nid=0x4340 runnable [0x00007fcbc94f3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058964c068> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a6fa058> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058928c438> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058928c438> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058928c5a0> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000 nid=0x433a waiting on condition [0x00007fcbc85c4000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a70a468> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000 nid=0x4336 runnable [0x00007fcbc6f90000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800 nid=0x4335 runnable [0x00007fcbc7091000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000 nid=0x4334 runnable [0x00007fcbc7392000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a796550> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000 nid=0x4333 runnable [0x00007fcbc7493000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000 nid=0x4332 runnable [0x00007fcbc7594000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition [0x00007fcbc7695000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition [0x00007fcbc7796000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800 nid=0x4318 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800 nid=0x4317 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000 nid=0x4316 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000 nid=0x4315 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000 nid=0x4314 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312 in Object.wait() [0x00007fcbe1be7000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800 nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on condition [0x00007fcbff9b9000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589585398> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000 nid=0x430b runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800 nid=0x430c runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800 nid=0x430d runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000 nid=0x430e runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000 nid=0x430f runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319 waiting on condition
>>> 
>>> JNI global references: 375
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> And the final worker, no active tasks:
>>> 
>>> 
>>> 2015-03-27 02:13:26
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000 nid=0x3c5b waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5 os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition [0x00007f36d2e33000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005895676e8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000 nid=0x3b88 runnable [0x00007f36dafb9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a86a7d8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800 nid=0x3b87 runnable [0x00007f36d7138000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058abea6c8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800 nid=0x3b86 runnable [0x00007f36daeb8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a86a898> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000 nid=0x3b85 runnable [0x00007f36d7e3b000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058abea788> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0 tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d waiting on condition [0x00007f36d7b3a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058aecd008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a runnable [0x00007f36db4be000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007f36f8025000 nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000 nid=0x3a96 runnable [0x00007f36dc0ea000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589202c58> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000 nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000 nid=0x3a6c runnable [0x00007f36dcb14000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a825c18> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005897334d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000005896b0d20> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000005896b0dc8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000 nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005895675a8> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058970f538> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800 nid=0x3a62 runnable [0x00007f36da5b1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800 nid=0x3a61 runnable [0x00007f36da6b2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000 nid=0x3a60 runnable [0x00007f36da7b3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800 nid=0x3a5f runnable [0x00007f36da8b4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800 nid=0x3a5e runnable [0x00007f36da9b5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition [0x00007f36dacb6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition [0x00007f36dadb7000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800 nid=0x3a43 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000 nid=0x3a42 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000 nid=0x3a41 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000 nid=0x3a40 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800 nid=0x3a3f runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d in Object.wait() [0x00007f36f51e5000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800 nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on condition [0x00007f3712fda000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a974ab8> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000 nid=0x3a34 runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800 nid=0x3a35 runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800 nid=0x3a36 runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000 nid=0x3a37 runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000 nid=0x3a3a runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44 waiting on condition
>>> 
>>> JNI global references: 486
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> And the driver:
>>> 
>>> 2015-03-27 02:20:47
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800 nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000 nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800 nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000 nid=0x41d0 waiting on condition [0x00007fd38feef000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0 tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000fd545eb8> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5 os_prio=0 tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition [0x00007fd3b8400000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000 nid=0x417f waiting on condition [0x00007fd398935000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800 nid=0x409c waiting on condition [0x00007fd3902f1000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000 nid=0x409b waiting on condition [0x00007fd3903f2000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800 nid=0x409a waiting on condition [0x00007fd3904f3000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800 nid=0x4098 waiting on condition [0x00007fd3905f4000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0 tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800 nid=0x4092 runnable [0x00007fd398b37000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e685f320> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800 nid=0x4091 runnable [0x00007fd398a36000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6871c30> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800 nid=0x4090 runnable [0x00007fd39903a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6873520> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0 tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0 tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e7f19070> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0 tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0 tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66fe220> (a java.util.concurrent.SynchronousQueue$TransferStack)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> 	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> 	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0 tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e740fa58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0 tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e740fce8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0 tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0 tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e737a0c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0 tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e737add8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0 tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0 tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>>> 
>>> "ResponseProcessor for block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable [0x00007fd39a941000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
>>> 	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>>> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>> 	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
>>> 	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
>>> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>>> 
>>> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000 nid=0x3fda waiting on condition [0x00007fd39aa42000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e58145f0> (a java.util.concurrent.Semaphore$NonfairSync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>>> 
>>> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000 nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>> 	at org.apache.spark.ContextCleaner.org <http://org.apache.spark.contextcleaner.org/>$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
>>> 	at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>>> 
>>> "DataStreamer for file /spark-logs/application_1427416187821_0004.inprogress block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait() [0x00007fd39ae44000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
>>> 	- locked <0x00000000e6994f30> (a java.util.LinkedList)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0 tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6995e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800 nid=0x3fb7 runnable [0x00007fd39b246000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6886b58> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Yarn application state monitor" #73 daemon prio=5 os_prio=0 tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>>> 
>>> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000 nid=0x3f77 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f waiting on condition [0x00007fd39b547000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e677ae98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "LeaseRenewer:hadoop@10.209.23.74:9000 <http://LeaseRenewer:hadoop@10.209.23.74:9000/>" #68 daemon prio=5 os_prio=0 tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e runnable [0x00007fd3b80ff000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "IPC Client (1489870911) connection to /10.209.23.74:9022 <http://10.209.23.74:9022/> from hadoop" #62 daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in Object.wait() [0x00007fd3b8501000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
>>> 	- locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
>>> 	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>>> 
>>> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0 tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9230> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
>>> 	at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
>>> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>>> 
>>> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in Object.wait() [0x00007fd3b8703000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e66e8190> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000 nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040 <http://SelectChannelConnector@0.0.0.0:4040/>" #54 daemon prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable [0x00007fd3b8fb6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>>> 	- locked <0x00000000e671a620> (a java.lang.Object)
>>> 	at org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0 tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>>> 	at org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0 tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e66609f0> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000 nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800 nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800 nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800 nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800 nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800 nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000 nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920 <http://SocketConnector@0.0.0.0:53920/>" #42 daemon prio=5 os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable [0x00007fd3ba0de000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
>>> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0 tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e6601270> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0 tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e6600ca8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0 tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e65380d8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "New I/O server boss #6" #36 daemon prio=5 os_prio=0 tid=0x00007fd39c04b800 nid=0x3df8 runnable [0x00007fd3ba6e2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800 nid=0x3df7 runnable [0x00007fd3ba7e3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800 nid=0x3df6 runnable [0x00007fd3ba8e4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000 nid=0x3df5 runnable [0x00007fd3ba9e5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800 nid=0x3df4 runnable [0x00007fd3baae6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000 nid=0x3df3 runnable [0x00007fd3babe7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5 os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition [0x00007fd3baef9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5 os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition [0x00007fd3baffa000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0 tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0 tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0 tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0 tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800 nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000 nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800 nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800 nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000 nid=0x3d72 waiting on condition [0x00007fd40c460000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000 nid=0x3d71 waiting on condition [0x00007fd40c561000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000 nid=0x3d70 waiting on condition [0x00007fd40c662000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131 <http://SocketConnector@0.0.0.0:53131/>" #14 daemon prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable [0x00007fd40c966000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
>>> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000 nid=0x3d67 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800 nid=0x3d66 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000 nid=0x3d65 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000 nid=0x3d64 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800 nid=0x3d63 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62 in Object.wait() [0x00007fd40f28e000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800 nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in Object.wait() [0x00007fd438220000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
>>> 	- locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
>>> 	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
>>> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
>>> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
>>> 	at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
>>> 	at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
>>> 	at org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
>>> 	at org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
>>> 	at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
>>> 	at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
>>> 	at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
>>> 	at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
>>> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
>>> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>>> 	at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
>>> 	at $line24.$read$$iwC$$iwC.<init>(<console>:35)
>>> 	at $line24.$read$$iwC.<init>(<console>:37)
>>> 	at $line24.$read.<init>(<console>:39)
>>> 	at $line24.$read$.<init>(<console>:43)
>>> 	at $line24.$read$.<clinit>(<console>)
>>> 	at $line24.$eval$.<init>(<console>:7)
>>> 	at $line24.$eval$.<clinit>(<console>)
>>> 	at $line24.$eval.$print(<console>)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:497)
>>> 	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>>> 	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>>> 	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>>> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>>> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>>> 	at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
>>> 	at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
>>> 	at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
>>> 	at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
>>> 	at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
>>> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>> 	at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>>> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>>> 	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
>>> 	at org.apache.spark.repl.Main$.main(Main.scala:31)
>>> 	at org.apache.spark.repl.Main.main(Main.scala)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:497)
>>> 	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>>> 	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>>> 	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>>> 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>>> 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>>> 
>>> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000 nid=0x3d5c runnable
>>> 
>>> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800 nid=0x3d5d runnable
>>> 
>>> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800 nid=0x3d5e runnable
>>> 
>>> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000 nid=0x3d5f runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68 waiting on condition
>>> 
>>> JNI global references: 420
>>> 
>>> 
>>> 
>>> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <michael@databricks.com <ma...@databricks.com>> wrote:
>>> Is it possible to jstack the executors and see where they are hanging?
>>> 
>>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jon.chase@gmail.com <ma...@gmail.com>> wrote:
>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB), executor memory 20GB, driver memory 10GB
>>> 
>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data spread out over roughly 2,000 Parquet files and my queries frequently hang. Simple queries like "select count(*) from ..." on the entire data set work ok.  Slightly more demanding ones with group by's and some aggregate functions (percentile_approx, avg, etc.) work ok as well, as long as I have some criteria in my where clause to keep the number of rows down.  
>>> 
>>> Once I hit some limit on query complexity and rows processed, my queries start to hang.  I've left them for up to an hour without seeing any progress.  No OOM's either - the job is just stuck.
>>> 
>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but with the same results: usually near the end of the tasks (like 780 of 800 complete), progress just stops:
>>> 
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in stage 1.0 (TID 1618) in 800 ms on ip-10-209-22-211.eu <http://ip-10-209-22-211.eu/>-west-1.compute.internal (748/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu <http://ip-10-105-12-41.eu/>-west-1.compute.internal (749/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu <http://ip-10-90-2-201.eu/>-west-1.compute.internal (750/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu <http://ip-10-90-2-201.eu/>-west-1.compute.internal (751/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu <http://ip-10-105-12-41.eu/>-west-1.compute.internal (752/800)
>>> 
>>> ^^^^^^^ this is where it stays forever
>>> 
>>> Looking at the Spark UI, several of the executors still list active tasks.  I do see that the Shuffle Read for executors that don't have any tasks remaining is around 100MB, whereas it's more like 10MB for the executors that still have tasks.  
>>> 
>>> The first stage, mapPartitions, always completes fine.  It's the second stage (takeOrdered), that hangs.
>>> 
>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also encountered it when using JSON files (instead of Parquet).
>>> 
>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do are having this issue.
>>> 
>>> 
>> 
> 
> 


Re: Spark SQL queries hang forever

Posted by Michael Allman <mi...@videoamp.com>.
Hi Reynold,

Our build is based off of the 1.3 branch as of commit 6d3c4d8b04b2738a821dfcc3df55a5635b89e506. Unless your fix is on another branch it appears this problem still exists.

LMK if there's anything else I can help with.

Thanks,

Michael

> On Apr 16, 2015, at 11:44 AM, Reynold Xin <rx...@databricks.com> wrote:
> 
> Thanks for the information.
> 
> Adding the dev list.
> 
> Do you mind trying 1.3.1 to see if the issue is fixed?  I could be wrong but it migtht be related to https://issues.apache.org/jira/browse/SPARK-6578 <https://issues.apache.org/jira/browse/SPARK-6578>
> 
>  
> 
> On Thu, Apr 16, 2015 at 2:34 AM, Michael Allman <michael@videoamp.com <ma...@videoamp.com>> wrote:
> After many, many manual (and automated) trials using a spark sql query that *sometimes* triggers the hang I found the following:
> 
> When I set spark.shuffle.blockTransferService to netty (the default), I can sometimes reproduce the problem.
> When I set spark.shuffle.blockTransferService to nio (the Spark 1.1 default), I have not yet been able to reproduce the problem.
> 
> While far from a reliable conclusion, the workaround I'm going with for now is setting "spark.shuffle.blockTransferService nio" in spark-defaults.conf. One of the reasons I homed in on this setting as a possible trigger was the fact that its default value changed starting in Spark 1.2. We haven't experienced a similar problem in our current Spark 1.1 cluster, and we're skipping directly to Spark 1.3. Has anyone else tried this apparent workaround?
> 
> Cheers,
> 
> Michael
> 
>> On Apr 15, 2015, at 11:23 PM, Michael Allman <michael@videoamp.com <ma...@videoamp.com>> wrote:
>> 
>> FWIW I'm attempting an upgrade from Spark 1.1 to Spark 1.3 and I'm seeing this exact problem in ShuffleBlockFetcherIterator in the query takeOrdered stage. Sometimes a single task hangs, sometimes a couple dozen. The final stage has 200 tasks. I'm using the Spark standalone scheduler, parquet files. Here's the stack trace of the stuck thread:
>> 
>> sun.misc.Unsafe.park(Native Method)
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:389)
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:154)
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:149)
>> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> org.apache.spark.scheduler.Task.run(Task.scala:64)
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> java.lang.Thread.run(Thread.java:745)
>> 
>> Anyone find a reliable fix/workaround? I'm going to try some things myself. If I find something that works I'll post back.
>> 
>> Cheers,
>> 
>> Michael
>> 
>> 
>>> On Mar 26, 2015, at 7:21 PM, Jon Chase <jon.chase@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> Here are the jstacks from a 3 worker cluster.  Two of the workers are sitting with 4 active tasks, the other has no active tasks.  Not sure if it makes a difference, but I only see an ExecutorLauncher instance via JPS on the worker that has no active tasks.  
>>> 
>>> Here's the first stack dump, this machine has 4 active tasks:
>>> 
>>> 2015-03-27 02:11:46
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000 nid=0x43a3 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5 os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition [0x00007f91bad9a000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884fc0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000 nid=0x42e2 runnable [0x00007f91b4c54000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005898aa000> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800 nid=0x42e1 runnable [0x00007f91b5d56000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058ad5b5f8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000 nid=0x42e0 runnable [0x00007f91b5e57000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005898aa0c0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800 nid=0x42df runnable [0x00007f91b5f58000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058ad5b6b8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cfda6800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0 tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2 waiting on condition [0x00007f91b6c5b000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058af8e0b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef runnable [0x00007f91baf9c000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0 tid=0x00007f91d8118800 nid=0x41ee waiting on condition [0x00007f91bae9b000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000 nid=0x41ed runnable [0x00007f91baa97000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589224af0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0 tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006c8837750> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0 tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cc53bff0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0 tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cfda69b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800 nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800 nid=0x41c6 runnable [0x00007f91bbac7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058984ee60> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0 tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005898f7290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0 tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058974cb00> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058974cb00> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0 tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x0000000589904038> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x0000000589904038> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0 tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884e80> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800 nid=0x41be waiting on condition [0x00007f91ba370000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O server boss #12" #39 daemon prio=5 os_prio=0 tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058992f188> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000 nid=0x41bc runnable [0x00007f91b9665000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589931218> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800 nid=0x41bb runnable [0x00007f91b9766000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800 nid=0x41ba runnable [0x00007f91b9867000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800 nid=0x41b9 runnable [0x00007f91b9968000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000 nid=0x41b8 runnable [0x00007f91b9c69000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5 os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition [0x00007f91b9d6a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5 os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition [0x00007f91b9e6b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800 nid=0x419e runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000 nid=0x419d waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000 nid=0x419c waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000 nid=0x419b waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800 nid=0x419a runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198 in Object.wait() [0x00007f91d4254000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800 nid=0x4197 in Object.wait() [0x00007f91d4355000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on condition [0x00007f91f1fdf000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589a1b5f0> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000 nid=0x4191 runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800 nid=0x4192 runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800 nid=0x4193 runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000 nid=0x4194 runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000 nid=0x4195 runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f waiting on condition
>>> 
>>> JNI global references: 319
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Here's the next worker, also for active tasks:
>>> 
>>> 2015-03-27 02:12:54
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000 nid=0x44f1 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5 os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition [0x00007fcbbf87f000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589510428> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0 tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006126cd358> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800 nid=0x4459 runnable [0x00007fcbc88c7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a6b8c98> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000 nid=0x4458 runnable [0x00007fcbc7897000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a6e5a18> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800 nid=0x4457 runnable [0x00007fcbc7998000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058aade110> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000 nid=0x4456 runnable [0x00007fcbc7b9a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058aade1d0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0 tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000615539dd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0 tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000617bd4238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006126cd538> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0 tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369 waiting on condition [0x00007fcbc4586000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058adabe70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366 runnable [0x00007fcbc7e9d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007fcbf9608800 nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800 nid=0x4364 runnable [0x00007fcbc8ac9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589116ae0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000 nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000 nid=0x4340 runnable [0x00007fcbc94f3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058964c068> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a6fa058> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058928c438> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058928c438> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058928c5a0> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000 nid=0x433a waiting on condition [0x00007fcbc85c4000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a70a468> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000 nid=0x4336 runnable [0x00007fcbc6f90000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800 nid=0x4335 runnable [0x00007fcbc7091000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000 nid=0x4334 runnable [0x00007fcbc7392000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a796550> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000 nid=0x4333 runnable [0x00007fcbc7493000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000 nid=0x4332 runnable [0x00007fcbc7594000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition [0x00007fcbc7695000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition [0x00007fcbc7796000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800 nid=0x4318 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800 nid=0x4317 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000 nid=0x4316 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000 nid=0x4315 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000 nid=0x4314 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312 in Object.wait() [0x00007fcbe1be7000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800 nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on condition [0x00007fcbff9b9000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589585398> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000 nid=0x430b runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800 nid=0x430c runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800 nid=0x430d runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000 nid=0x430e runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000 nid=0x430f runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319 waiting on condition
>>> 
>>> JNI global references: 375
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> And the final worker, no active tasks:
>>> 
>>> 
>>> 2015-03-27 02:13:26
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000 nid=0x3c5b waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5 os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition [0x00007f36d2e33000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005895676e8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000 nid=0x3b88 runnable [0x00007f36dafb9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a86a7d8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800 nid=0x3b87 runnable [0x00007f36d7138000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058abea6c8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800 nid=0x3b86 runnable [0x00007f36daeb8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a86a898> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000 nid=0x3b85 runnable [0x00007f36d7e3b000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058abea788> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0 tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d waiting on condition [0x00007f36d7b3a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058aecd008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a runnable [0x00007f36db4be000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007f36f8025000 nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000 nid=0x3a96 runnable [0x00007f36dc0ea000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589202c58> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000 nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000 nid=0x3a6c runnable [0x00007f36dcb14000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a825c18> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005897334d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000005896b0d20> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000005896b0dc8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000 nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005895675a8> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058970f538> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800 nid=0x3a62 runnable [0x00007f36da5b1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800 nid=0x3a61 runnable [0x00007f36da6b2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000 nid=0x3a60 runnable [0x00007f36da7b3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800 nid=0x3a5f runnable [0x00007f36da8b4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800 nid=0x3a5e runnable [0x00007f36da9b5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition [0x00007f36dacb6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition [0x00007f36dadb7000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800 nid=0x3a43 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000 nid=0x3a42 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000 nid=0x3a41 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000 nid=0x3a40 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800 nid=0x3a3f runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d in Object.wait() [0x00007f36f51e5000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800 nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on condition [0x00007f3712fda000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a974ab8> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000 nid=0x3a34 runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800 nid=0x3a35 runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800 nid=0x3a36 runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000 nid=0x3a37 runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000 nid=0x3a3a runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44 waiting on condition
>>> 
>>> JNI global references: 486
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> And the driver:
>>> 
>>> 2015-03-27 02:20:47
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800 nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000 nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800 nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000 nid=0x41d0 waiting on condition [0x00007fd38feef000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0 tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000fd545eb8> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5 os_prio=0 tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition [0x00007fd3b8400000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000 nid=0x417f waiting on condition [0x00007fd398935000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800 nid=0x409c waiting on condition [0x00007fd3902f1000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000 nid=0x409b waiting on condition [0x00007fd3903f2000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800 nid=0x409a waiting on condition [0x00007fd3904f3000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800 nid=0x4098 waiting on condition [0x00007fd3905f4000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0 tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800 nid=0x4092 runnable [0x00007fd398b37000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e685f320> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800 nid=0x4091 runnable [0x00007fd398a36000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6871c30> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800 nid=0x4090 runnable [0x00007fd39903a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6873520> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0 tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0 tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e7f19070> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0 tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0 tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66fe220> (a java.util.concurrent.SynchronousQueue$TransferStack)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> 	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> 	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0 tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e740fa58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0 tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e740fce8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0 tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0 tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e737a0c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0 tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e737add8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0 tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0 tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>>> 
>>> "ResponseProcessor for block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable [0x00007fd39a941000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
>>> 	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>>> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>> 	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
>>> 	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
>>> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>>> 
>>> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000 nid=0x3fda waiting on condition [0x00007fd39aa42000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e58145f0> (a java.util.concurrent.Semaphore$NonfairSync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>>> 
>>> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000 nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>> 	at org.apache.spark.ContextCleaner.org <http://org.apache.spark.contextcleaner.org/>$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
>>> 	at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>>> 
>>> "DataStreamer for file /spark-logs/application_1427416187821_0004.inprogress block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait() [0x00007fd39ae44000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
>>> 	- locked <0x00000000e6994f30> (a java.util.LinkedList)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0 tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6995e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800 nid=0x3fb7 runnable [0x00007fd39b246000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6886b58> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Yarn application state monitor" #73 daemon prio=5 os_prio=0 tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>>> 
>>> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000 nid=0x3f77 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f waiting on condition [0x00007fd39b547000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e677ae98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "LeaseRenewer:hadoop@10.209.23.74:9000 <http://LeaseRenewer:hadoop@10.209.23.74:9000/>" #68 daemon prio=5 os_prio=0 tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e runnable [0x00007fd3b80ff000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "IPC Client (1489870911) connection to /10.209.23.74:9022 <http://10.209.23.74:9022/> from hadoop" #62 daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in Object.wait() [0x00007fd3b8501000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
>>> 	- locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
>>> 	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>>> 
>>> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0 tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9230> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
>>> 	at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
>>> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>>> 
>>> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in Object.wait() [0x00007fd3b8703000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e66e8190> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000 nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040 <http://SelectChannelConnector@0.0.0.0:4040/>" #54 daemon prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable [0x00007fd3b8fb6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>>> 	- locked <0x00000000e671a620> (a java.lang.Object)
>>> 	at org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0 tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>>> 	at org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0 tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e66609f0> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000 nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800 nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800 nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800 nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800 nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800 nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000 nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920 <http://SocketConnector@0.0.0.0:53920/>" #42 daemon prio=5 os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable [0x00007fd3ba0de000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
>>> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0 tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e6601270> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0 tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e6600ca8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0 tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e65380d8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "New I/O server boss #6" #36 daemon prio=5 os_prio=0 tid=0x00007fd39c04b800 nid=0x3df8 runnable [0x00007fd3ba6e2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800 nid=0x3df7 runnable [0x00007fd3ba7e3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800 nid=0x3df6 runnable [0x00007fd3ba8e4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000 nid=0x3df5 runnable [0x00007fd3ba9e5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800 nid=0x3df4 runnable [0x00007fd3baae6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000 nid=0x3df3 runnable [0x00007fd3babe7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5 os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition [0x00007fd3baef9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5 os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition [0x00007fd3baffa000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0 tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0 tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0 tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0 tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800 nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000 nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800 nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800 nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000 nid=0x3d72 waiting on condition [0x00007fd40c460000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000 nid=0x3d71 waiting on condition [0x00007fd40c561000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000 nid=0x3d70 waiting on condition [0x00007fd40c662000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131 <http://SocketConnector@0.0.0.0:53131/>" #14 daemon prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable [0x00007fd40c966000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
>>> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000 nid=0x3d67 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800 nid=0x3d66 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000 nid=0x3d65 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000 nid=0x3d64 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800 nid=0x3d63 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62 in Object.wait() [0x00007fd40f28e000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800 nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in Object.wait() [0x00007fd438220000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
>>> 	- locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
>>> 	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
>>> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
>>> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
>>> 	at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
>>> 	at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
>>> 	at org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
>>> 	at org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
>>> 	at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
>>> 	at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
>>> 	at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
>>> 	at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
>>> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
>>> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>>> 	at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
>>> 	at $line24.$read$$iwC$$iwC.<init>(<console>:35)
>>> 	at $line24.$read$$iwC.<init>(<console>:37)
>>> 	at $line24.$read.<init>(<console>:39)
>>> 	at $line24.$read$.<init>(<console>:43)
>>> 	at $line24.$read$.<clinit>(<console>)
>>> 	at $line24.$eval$.<init>(<console>:7)
>>> 	at $line24.$eval$.<clinit>(<console>)
>>> 	at $line24.$eval.$print(<console>)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:497)
>>> 	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>>> 	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>>> 	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>>> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>>> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>>> 	at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
>>> 	at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
>>> 	at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
>>> 	at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
>>> 	at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
>>> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>> 	at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>>> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>>> 	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
>>> 	at org.apache.spark.repl.Main$.main(Main.scala:31)
>>> 	at org.apache.spark.repl.Main.main(Main.scala)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:497)
>>> 	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>>> 	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>>> 	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>>> 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>>> 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>>> 
>>> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000 nid=0x3d5c runnable
>>> 
>>> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800 nid=0x3d5d runnable
>>> 
>>> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800 nid=0x3d5e runnable
>>> 
>>> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000 nid=0x3d5f runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68 waiting on condition
>>> 
>>> JNI global references: 420
>>> 
>>> 
>>> 
>>> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <michael@databricks.com <ma...@databricks.com>> wrote:
>>> Is it possible to jstack the executors and see where they are hanging?
>>> 
>>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jon.chase@gmail.com <ma...@gmail.com>> wrote:
>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB), executor memory 20GB, driver memory 10GB
>>> 
>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data spread out over roughly 2,000 Parquet files and my queries frequently hang. Simple queries like "select count(*) from ..." on the entire data set work ok.  Slightly more demanding ones with group by's and some aggregate functions (percentile_approx, avg, etc.) work ok as well, as long as I have some criteria in my where clause to keep the number of rows down.  
>>> 
>>> Once I hit some limit on query complexity and rows processed, my queries start to hang.  I've left them for up to an hour without seeing any progress.  No OOM's either - the job is just stuck.
>>> 
>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but with the same results: usually near the end of the tasks (like 780 of 800 complete), progress just stops:
>>> 
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in stage 1.0 (TID 1618) in 800 ms on ip-10-209-22-211.eu <http://ip-10-209-22-211.eu/>-west-1.compute.internal (748/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu <http://ip-10-105-12-41.eu/>-west-1.compute.internal (749/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu <http://ip-10-90-2-201.eu/>-west-1.compute.internal (750/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu <http://ip-10-90-2-201.eu/>-west-1.compute.internal (751/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu <http://ip-10-105-12-41.eu/>-west-1.compute.internal (752/800)
>>> 
>>> ^^^^^^^ this is where it stays forever
>>> 
>>> Looking at the Spark UI, several of the executors still list active tasks.  I do see that the Shuffle Read for executors that don't have any tasks remaining is around 100MB, whereas it's more like 10MB for the executors that still have tasks.  
>>> 
>>> The first stage, mapPartitions, always completes fine.  It's the second stage (takeOrdered), that hangs.
>>> 
>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also encountered it when using JSON files (instead of Parquet).
>>> 
>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do are having this issue.
>>> 
>>> 
>> 
> 
> 


Re: Spark SQL queries hang forever

Posted by Reynold Xin <rx...@databricks.com>.
Thanks for the information.

Adding the dev list.

Do you mind trying 1.3.1 to see if the issue is fixed?  I could be wrong
but it migtht be related to https://issues.apache.org/jira/browse/SPARK-6578



On Thu, Apr 16, 2015 at 2:34 AM, Michael Allman <mi...@videoamp.com>
wrote:

> After many, many manual (and automated) trials using a spark sql query
> that *sometimes* triggers the hang I found the following:
>
>
>    - When I set spark.shuffle.blockTransferService to netty (the
>    default), I can sometimes reproduce the problem.
>    - When I set spark.shuffle.blockTransferService to nio (the Spark 1.1
>    default), I have not yet been able to reproduce the problem.
>
>
> While far from a reliable conclusion, the workaround I'm going with for
> now is setting "spark.shuffle.blockTransferService nio" in
> spark-defaults.conf. One of the reasons I homed in on this setting as a
> possible trigger was the fact that its default value changed starting in
> Spark 1.2. We haven't experienced a similar problem in our current Spark
> 1.1 cluster, and we're skipping directly to Spark 1.3. Has anyone else
> tried this apparent workaround?
>
> Cheers,
>
> Michael
>
> On Apr 15, 2015, at 11:23 PM, Michael Allman <mi...@videoamp.com> wrote:
>
> FWIW I'm attempting an upgrade from Spark 1.1 to Spark 1.3 and I'm seeing
> this exact problem in ShuffleBlockFetcherIterator in the query takeOrdered
> stage. Sometimes a single task hangs, sometimes a couple dozen. The final
> stage has 200 tasks. I'm using the Spark standalone scheduler, parquet
> files. Here's the stack trace of the stuck thread:
>
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:389)
>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:154)
>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:149)
> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> org.apache.spark.scheduler.Task.run(Task.scala:64)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
>
> Anyone find a reliable fix/workaround? I'm going to try some things
> myself. If I find something that works I'll post back.
>
> Cheers,
>
> Michael
>
>
> On Mar 26, 2015, at 7:21 PM, Jon Chase <jo...@gmail.com> wrote:
>
> Here are the jstacks from a 3 worker cluster.  Two of the workers are
> sitting with 4 active tasks, the other has no active tasks.  Not sure if it
> makes a difference, but I only see an ExecutorLauncher instance via JPS on
> the worker that has no active tasks.
>
> Here's the first stack dump, this machine has 4 active tasks:
>
> 2015-03-27 02:11:46
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
> nid=0x43a3 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
> os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
> [0x00007f91bad9a000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884fc0> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
> nid=0x42e2 runnable [0x00007f91b4c54000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005898aa000> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
> nid=0x42e1 runnable [0x00007f91b5d56000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058ad5b5f8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
> nid=0x42e0 runnable [0x00007f91b5e57000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005898aa0c0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
> nid=0x42df runnable [0x00007f91b5f58000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058ad5b6b8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cfda6800> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
> tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
> waiting on condition [0x00007f91b6c5b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058af8e0b8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
> runnable [0x00007f91baf9c000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0
> tid=0x00007f91d8118800 nid=0x41ee waiting on condition [0x00007f91bae9b000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
> nid=0x41ed runnable [0x00007f91baa97000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589224af0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
> tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006c8837750> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
> tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cc53bff0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
> tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cfda69b0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
> nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
> nid=0x41c6 runnable [0x00007f91bbac7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058984ee60> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
> tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005898f7290> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
> tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058974cb00> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
> tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000589904038> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x0000000589904038> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
> tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884e80> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800
> nid=0x41be waiting on condition [0x00007f91ba370000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O server boss #12" #39 daemon prio=5 os_prio=0
> tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
> - locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
> nid=0x41bc runnable [0x00007f91b9665000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
> - locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
> nid=0x41bb runnable [0x00007f91b9766000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
> - locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
> nid=0x41ba runnable [0x00007f91b9867000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
> nid=0x41b9 runnable [0x00007f91b9968000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
> - locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
> nid=0x41b8 runnable [0x00007f91b9c69000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
> - locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5
> os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
> [0x00007f91b9d6a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884f20> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5
> os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
> [0x00007f91b9e6b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884f20> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
> nid=0x419e runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
> nid=0x419d waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
> nid=0x419c waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
> nid=0x419b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
> nid=0x419a runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198
> in Object.wait() [0x00007f91d4254000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
> nid=0x4197 in Object.wait() [0x00007f91d4355000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
> condition [0x00007f91f1fdf000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589a1b5f0> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
> nid=0x4191 runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
> nid=0x4192 runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
> nid=0x4193 runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
> nid=0x4194 runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
> nid=0x4195 runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
> waiting on condition
>
> JNI global references: 319
>
>
>
>
>
>
>
>
>
>
>
>
> Here's the next worker, also for active tasks:
>
> 2015-03-27 02:12:54
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
> nid=0x44f1 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
> os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
> [0x00007fcbbf87f000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589510428> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
> tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006126cd358> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
> nid=0x4459 runnable [0x00007fcbc88c7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a6b8c98> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
> nid=0x4458 runnable [0x00007fcbc7897000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a6e5a18> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
> nid=0x4457 runnable [0x00007fcbc7998000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058aade110> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
> nid=0x4456 runnable [0x00007fcbc7b9a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058aade1d0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
> tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000615539dd8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
> tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000617bd4238> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006126cd538> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
> tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
> waiting on condition [0x00007fcbc4586000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058adabe70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
> runnable [0x00007fcbc7e9d000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> tid=0x00007fcbf9608800 nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
> nid=0x4364 runnable [0x00007fcbc8ac9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589116ae0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
> nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
> nid=0x4340 runnable [0x00007fcbc94f3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058964c068> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a6fa058> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058928c438> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058928c438> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058928c5a0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000
> nid=0x433a waiting on condition [0x00007fcbc85c4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a70a468> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
> - locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
> nid=0x4336 runnable [0x00007fcbc6f90000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
> - locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
> nid=0x4335 runnable [0x00007fcbc7091000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
> - locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
> nid=0x4334 runnable [0x00007fcbc7392000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
> - locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
> nid=0x4333 runnable [0x00007fcbc7493000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
> - locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
> nid=0x4332 runnable [0x00007fcbc7594000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
> - locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
> [0x00007fcbc7695000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005893c1820> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
> [0x00007fcbc7796000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005893c1820> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
> nid=0x4318 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
> nid=0x4317 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
> nid=0x4316 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
> nid=0x4315 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
> nid=0x4314 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312
> in Object.wait() [0x00007fcbe1be7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
> nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
> condition [0x00007fcbff9b9000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589585398> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
> nid=0x430b runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
> nid=0x430c runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
> nid=0x430d runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
> nid=0x430e runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
> nid=0x430f runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
> waiting on condition
>
> JNI global references: 375
>
>
>
>
>
>
>
>
>
>
> And the final worker, no active tasks:
>
>
> 2015-03-27 02:13:26
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
> nid=0x3c5b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
> os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
> [0x00007f36d2e33000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005895676e8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
> nid=0x3b88 runnable [0x00007f36dafb9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a86a7d8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
> nid=0x3b87 runnable [0x00007f36d7138000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058abea6c8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
> nid=0x3b86 runnable [0x00007f36daeb8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a86a898> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
> nid=0x3b85 runnable [0x00007f36d7e3b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058abea788> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
> tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
> waiting on condition [0x00007f36d7b3a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058aecd008> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
> runnable [0x00007f36db4be000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> tid=0x00007f36f8025000 nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
> nid=0x3a96 runnable [0x00007f36dc0ea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589202c58> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
> nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
> nid=0x3a6c runnable [0x00007f36dcb14000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a825c18> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005897334d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000005896b0d20> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000
> nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005895675a8> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
> - locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
> nid=0x3a62 runnable [0x00007f36da5b1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
> - locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
> nid=0x3a61 runnable [0x00007f36da6b2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
> - locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
> nid=0x3a60 runnable [0x00007f36da7b3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
> - locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
> nid=0x3a5f runnable [0x00007f36da8b4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
> - locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
> nid=0x3a5e runnable [0x00007f36da9b5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
> - locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
> [0x00007f36dacb6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589567648> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
> [0x00007f36dadb7000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589567648> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
> nid=0x3a43 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
> nid=0x3a42 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
> nid=0x3a41 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
> nid=0x3a40 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
> nid=0x3a3f runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d
> in Object.wait() [0x00007f36f51e5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
> nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
> condition [0x00007f3712fda000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a974ab8> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
> nid=0x3a34 runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
> nid=0x3a35 runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
> nid=0x3a36 runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
> nid=0x3a37 runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
> nid=0x3a3a runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
> waiting on condition
>
> JNI global references: 486
>
>
>
>
>
>
>
>
> And the driver:
>
> 2015-03-27 02:20:47
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
> nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
> nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
> nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
> nid=0x41d0 waiting on condition [0x00007fd38feef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
> tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000fd545eb8> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5
> os_prio=0 tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition
> [0x00007fd3b8400000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
> nid=0x417f waiting on condition [0x00007fd398935000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800
> nid=0x409c waiting on condition [0x00007fd3902f1000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000
> nid=0x409b waiting on condition [0x00007fd3903f2000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800
> nid=0x409a waiting on condition [0x00007fd3904f3000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800
> nid=0x4098 waiting on condition [0x00007fd3905f4000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
> tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
> nid=0x4092 runnable [0x00007fd398b37000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e685f320> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
> nid=0x4091 runnable [0x00007fd398a36000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6871c30> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
> nid=0x4090 runnable [0x00007fd39903a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6873520> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
> tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
> tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e7f19070> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0
> tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
> tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66fe220> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
> tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e740fa58> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
> tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e740fce8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
> tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
> tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e737a0c8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
> tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e737add8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
> tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> at java.lang.Thread.run(Thread.java:745)
>
> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
> tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
> at
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>
> "ResponseProcessor for block
> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
> prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
> [0x00007fd39a941000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
> - locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>
> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
> nid=0x3fda waiting on condition [0x00007fd39aa42000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e58145f0> (a
> java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>
> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000
> nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at org.apache.spark.ContextCleaner.org
> <http://org.apache.spark.contextcleaner.org/>
> $apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
> at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>
> "DataStreamer for file
> /spark-logs/application_1427416187821_0004.inprogress block
> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon
> prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait()
> [0x00007fd39ae44000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
> - locked <0x00000000e6994f30> (a java.util.LinkedList)
>
> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
> tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6995e88> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
> nid=0x3fb7 runnable [0x00007fd39b246000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6886b58> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Yarn application state monitor" #73 daemon prio=5 os_prio=0
> tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>
> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
> nid=0x3f77 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f
> waiting on condition [0x00007fd39b547000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e677ae98> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
> tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e
> runnable [0x00007fd3b80ff000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop"
> #62 daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in
> Object.wait() [0x00007fd3b8501000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
> - locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>
> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
> tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9230> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> at
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>
> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in
> Object.wait() [0x00007fd3b8703000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e66e8190> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
> nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54
> daemon prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
> [0x00007fd3b8fb6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> - locked <0x00000000e671a620> (a java.lang.Object)
> at
> org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
> tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> at
> org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
> tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e66609f0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
> nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
> nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
> nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
> nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
> nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
> nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
> nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon
> prio=5 os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable
> [0x00007fd3ba0de000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
> tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e6601270> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
> tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
> tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e65380d8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "New I/O server boss #6" #36 daemon prio=5 os_prio=0
> tid=0x00007fd39c04b800 nid=0x3df8 runnable [0x00007fd3ba6e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
> nid=0x3df7 runnable [0x00007fd3ba7e3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
> nid=0x3df6 runnable [0x00007fd3ba8e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
> nid=0x3df5 runnable [0x00007fd3ba9e5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
> - locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
> nid=0x3df4 runnable [0x00007fd3baae6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
> - locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
> nid=0x3df3 runnable [0x00007fd3babe7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
> - locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
> os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
> [0x00007fd3baef9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64bbb40> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
> os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
> [0x00007fd3baffa000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64bbb40> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0
> tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0
> tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0
> tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
> tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
> nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
> nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
> nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
> nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
> nid=0x3d72 waiting on condition [0x00007fd40c460000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
> nid=0x3d71 waiting on condition [0x00007fd40c561000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
> nid=0x3d70 waiting on condition [0x00007fd40c662000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon
> prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable
> [0x00007fd40c966000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
> nid=0x3d67 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
> nid=0x3d66 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
> nid=0x3d65 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
> nid=0x3d64 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
> nid=0x3d63 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62
> in Object.wait() [0x00007fd40f28e000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
> nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
> Object.wait() [0x00007fd438220000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> - locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
> at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
> at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
> at
> org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
> at
> org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
> at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
> at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
> at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
> at
> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
> at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
> at $line24.$read$$iwC$$iwC.<init>(<console>:35)
> at $line24.$read$$iwC.<init>(<console>:37)
> at $line24.$read.<init>(<console>:39)
> at $line24.$read$.<init>(<console>:43)
> at $line24.$read$.<clinit>(<console>)
> at $line24.$eval$.<init>(<console>:7)
> at $line24.$eval$.<clinit>(<console>)
> at $line24.$eval.$print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
> at org.apache.spark.repl.SparkILoop.org
> <http://org.apache.spark.repl.sparkiloop.org/>
> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.org
> <http://org.apache.spark.repl.sparkiloop.org/>
> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000
> nid=0x3d5c runnable
>
> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800
> nid=0x3d5d runnable
>
> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800
> nid=0x3d5e runnable
>
> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000
> nid=0x3d5f runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
> waiting on condition
>
> JNI global references: 420
>
>
>
> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> Is it possible to jstack the executors and see where they are hanging?
>>
>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:
>>
>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB),
>>> executor memory 20GB, driver memory 10GB
>>>
>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data
>>> spread out over roughly 2,000 Parquet files and my queries frequently hang.
>>> Simple queries like "select count(*) from ..." on the entire data set work
>>> ok.  Slightly more demanding ones with group by's and some aggregate
>>> functions (percentile_approx, avg, etc.) work ok as well, as long as I have
>>> some criteria in my where clause to keep the number of rows down.
>>>
>>> Once I hit some limit on query complexity and rows processed, my queries
>>> start to hang.  I've left them for up to an hour without seeing any
>>> progress.  No OOM's either - the job is just stuck.
>>>
>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but
>>> with the same results: usually near the end of the tasks (like 780 of 800
>>> complete), progress just stops:
>>>
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in
>>> stage 1.0 (TID 1618) in 800 ms on ip-10-209-22-211.eu-west-1.compute.internal
>>> (748/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in
>>> stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>> (749/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in
>>> stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>> (750/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in
>>> stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>> (751/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in
>>> stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>> (752/800)
>>>
>>> ^^^^^^^ this is where it stays forever
>>>
>>> Looking at the Spark UI, several of the executors still list active
>>> tasks.  I do see that the Shuffle Read for executors that don't have any
>>> tasks remaining is around 100MB, whereas it's more like 10MB for the
>>> executors that still have tasks.
>>>
>>> The first stage, mapPartitions, always completes fine.  It's the second
>>> stage (takeOrdered), that hangs.
>>>
>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
>>> encountered it when using JSON files (instead of Parquet).
>>>
>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
>>> are having this issue.
>>>
>>
>>
>
>
>

Re: Spark SQL queries hang forever

Posted by Zsolt Tóth <to...@gmail.com>.
Thanks Michael, setting spark.shuffle.blockTransferService=nio fixed my
issue too.

2015-04-16 17:17 GMT+02:00 Jon Chase <jo...@gmail.com>:

> Michael - nice find!
>
> I can confirm that using nio for the block transfer service fixes the
> issue:
>
>     --conf "spark.shuffle.blockTransferService=nio"
>
> (note: setting this using sc.getConf(...) after the repl has started will
> not work)
>
> I've reported this as a bug:
>
> https://issues.apache.org/jira/browse/SPARK-6962
>
> Please vote for this in Jira if you're having the same issue.
>
>
> On Thu, Apr 16, 2015 at 3:34 AM, Michael Allman <mi...@videoamp.com>
> wrote:
>
>> After many, many manual (and automated) trials using a spark sql query
>> that *sometimes* triggers the hang I found the following:
>>
>>
>>    - When I set spark.shuffle.blockTransferService to netty (the
>>    default), I can sometimes reproduce the problem.
>>    - When I set spark.shuffle.blockTransferService to nio (the Spark 1.1
>>    default), I have not yet been able to reproduce the problem.
>>
>>
>> While far from a reliable conclusion, the workaround I'm going with for
>> now is setting "spark.shuffle.blockTransferService nio" in
>> spark-defaults.conf. One of the reasons I homed in on this setting as a
>> possible trigger was the fact that its default value changed starting in
>> Spark 1.2. We haven't experienced a similar problem in our current Spark
>> 1.1 cluster, and we're skipping directly to Spark 1.3. Has anyone else
>> tried this apparent workaround?
>>
>> Cheers,
>>
>> Michael
>>
>> On Apr 15, 2015, at 11:23 PM, Michael Allman <mi...@videoamp.com>
>> wrote:
>>
>> FWIW I'm attempting an upgrade from Spark 1.1 to Spark 1.3 and I'm seeing
>> this exact problem in ShuffleBlockFetcherIterator in the query takeOrdered
>> stage. Sometimes a single task hangs, sometimes a couple dozen. The final
>> stage has 200 tasks. I'm using the Spark standalone scheduler, parquet
>> files. Here's the stack trace of the stuck thread:
>>
>> sun.misc.Unsafe.park(Native Method)
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:389)
>>
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>>
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:154)
>>
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:149)
>> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> org.apache.spark.scheduler.Task.run(Task.scala:64)
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> java.lang.Thread.run(Thread.java:745)
>>
>> Anyone find a reliable fix/workaround? I'm going to try some things
>> myself. If I find something that works I'll post back.
>>
>> Cheers,
>>
>> Michael
>>
>>
>> On Mar 26, 2015, at 7:21 PM, Jon Chase <jo...@gmail.com> wrote:
>>
>> Here are the jstacks from a 3 worker cluster.  Two of the workers are
>> sitting with 4 active tasks, the other has no active tasks.  Not sure if it
>> makes a difference, but I only see an ExecutorLauncher instance via JPS on
>> the worker that has no active tasks.
>>
>> Here's the first stack dump, this machine has 4 active tasks:
>>
>> 2015-03-27 02:11:46
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>
>> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
>> nid=0x43a3 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
>> os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
>> [0x00007f91bad9a000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000589884fc0> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
>> nid=0x42e2 runnable [0x00007f91b4c54000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000005898aa000> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
>> nid=0x42e1 runnable [0x00007f91b5d56000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058ad5b5f8> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
>> nid=0x42e0 runnable [0x00007f91b5e57000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000005898aa0c0> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
>> nid=0x42df runnable [0x00007f91b5f58000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058ad5b6b8> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
>> tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000006cfda6800> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> at
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> at
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
>> tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
>> waiting on condition [0x00007f91b6c5b000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000058af8e0b8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
>> runnable [0x00007f91baf9c000]
>>    java.lang.Thread.State: RUNNABLE
>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> at
>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> at
>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0
>> tid=0x00007f91d8118800 nid=0x41ee waiting on condition [0x00007f91bae9b000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> at
>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
>> nid=0x41ed runnable [0x00007f91baa97000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x0000000589224af0> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
>> tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000006c8837750> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> at
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> at
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
>> tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000006cc53bff0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> at
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> at
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
>> tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000006cfda69b0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> at
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> at
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
>> nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>
>> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
>> nid=0x41c6 runnable [0x00007f91bbac7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058984ee60> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
>> tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005898f7290> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
>> tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
>> at java.lang.Object.wait(Object.java:502)
>> at java.util.TimerThread.mainLoop(Timer.java:526)
>> - locked <0x000000058974cb00> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
>> tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x0000000589904038> (a java.util.TaskQueue)
>> at java.lang.Object.wait(Object.java:502)
>> at java.util.TimerThread.mainLoop(Timer.java:526)
>> - locked <0x0000000589904038> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
>> tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000589884e80> (a
>> scala.concurrent.forkjoin.ForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0
>> tid=0x00007f91ed92c800 nid=0x41be waiting on condition [0x00007f91ba370000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>> at
>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O server boss #12" #39 daemon prio=5 os_prio=0
>> tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
>> - locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> at
>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
>> nid=0x41bc runnable [0x00007f91b9665000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
>> - locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
>> nid=0x41bb runnable [0x00007f91b9766000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
>> - locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
>> nid=0x41ba runnable [0x00007f91b9867000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
>> nid=0x41b9 runnable [0x00007f91b9968000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
>> - locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
>> nid=0x41b8 runnable [0x00007f91b9c69000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5
>> os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
>> [0x00007f91b9d6a000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000589884f20> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5
>> os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
>> [0x00007f91b9e6b000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000589884f20> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>> tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> at
>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
>> nid=0x419e runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
>> nid=0x419d waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
>> nid=0x419c waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
>> nid=0x419b waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
>> nid=0x419a runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>> tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198
>> in Object.wait() [0x00007f91d4254000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>
>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
>> nid=0x4197 in Object.wait() [0x00007f91d4355000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>>
>> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
>> condition [0x00007f91f1fdf000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000589a1b5f0> (a
>> java.util.concurrent.CountDownLatch$Sync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>> at
>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>> at
>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> at
>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> at scala.concurrent.Await$.ready(package.scala:86)
>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>> at
>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>> at
>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> at
>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>
>> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>>
>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
>> nid=0x4191 runnable
>>
>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
>> nid=0x4192 runnable
>>
>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
>> nid=0x4193 runnable
>>
>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
>> nid=0x4194 runnable
>>
>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
>> nid=0x4195 runnable
>>
>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
>> waiting on condition
>>
>> JNI global references: 319
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Here's the next worker, also for active tasks:
>>
>> 2015-03-27 02:12:54
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>
>> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
>> nid=0x44f1 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
>> os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
>> [0x00007fcbbf87f000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000589510428> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
>> tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000006126cd358> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> at
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> at
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
>> nid=0x4459 runnable [0x00007fcbc88c7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058a6b8c98> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
>> nid=0x4458 runnable [0x00007fcbc7897000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058a6e5a18> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
>> nid=0x4457 runnable [0x00007fcbc7998000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058aade110> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
>> nid=0x4456 runnable [0x00007fcbc7b9a000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058aade1d0> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
>> tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000615539dd8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> at
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> at
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
>> tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000617bd4238> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> at
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> at
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
>> tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000006126cd538> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> at
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> at
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> at
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> at
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
>> tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
>> waiting on condition [0x00007fcbc4586000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000058adabe70> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
>> runnable [0x00007fcbc7e9d000]
>>    java.lang.Thread.State: RUNNABLE
>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> at
>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> at
>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
>> tid=0x00007fcbf9608800 nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> at
>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
>> nid=0x4364 runnable [0x00007fcbc8ac9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x0000000589116ae0> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
>> nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>
>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
>> nid=0x4340 runnable [0x00007fcbc94f3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058964c068> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
>> tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000058a6fa058> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
>> tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x000000058928c438> (a java.util.TaskQueue)
>> at java.lang.Object.wait(Object.java:502)
>> at java.util.TimerThread.mainLoop(Timer.java:526)
>> - locked <0x000000058928c438> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
>> tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
>> at java.lang.Object.wait(Object.java:502)
>> at java.util.TimerThread.mainLoop(Timer.java:526)
>> - locked <0x000000058928c5a0> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
>> tid=0x00007fcbdc29d000 nid=0x433a waiting on condition [0x00007fcbc85c4000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>> at
>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
>> tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000058a70a468> (a
>> scala.concurrent.forkjoin.ForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
>> tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
>> - locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> at
>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
>> nid=0x4336 runnable [0x00007fcbc6f90000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
>> - locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
>> nid=0x4335 runnable [0x00007fcbc7091000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
>> - locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
>> nid=0x4334 runnable [0x00007fcbc7392000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
>> - locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
>> nid=0x4333 runnable [0x00007fcbc7493000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
>> - locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
>> nid=0x4332 runnable [0x00007fcbc7594000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
>> - locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
>> os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
>> [0x00007fcbc7695000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005893c1820> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
>> os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
>> [0x00007fcbc7796000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005893c1820> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>> tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> at
>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
>> nid=0x4318 runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
>> nid=0x4317 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
>> nid=0x4316 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
>> nid=0x4315 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
>> nid=0x4314 runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>> tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312
>> in Object.wait() [0x00007fcbe1be7000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>
>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
>> nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>>
>> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
>> condition [0x00007fcbff9b9000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000589585398> (a
>> java.util.concurrent.CountDownLatch$Sync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>> at
>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>> at
>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> at
>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> at scala.concurrent.Await$.ready(package.scala:86)
>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>> at
>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>> at
>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> at
>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>
>> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>>
>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
>> nid=0x430b runnable
>>
>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
>> nid=0x430c runnable
>>
>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
>> nid=0x430d runnable
>>
>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
>> nid=0x430e runnable
>>
>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
>> nid=0x430f runnable
>>
>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
>> waiting on condition
>>
>> JNI global references: 375
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> And the final worker, no active tasks:
>>
>>
>> 2015-03-27 02:13:26
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>
>> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
>> nid=0x3c5b waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
>> os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
>> [0x00007f36d2e33000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005895676e8> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
>> nid=0x3b88 runnable [0x00007f36dafb9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058a86a7d8> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
>> nid=0x3b87 runnable [0x00007f36d7138000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058abea6c8> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
>> nid=0x3b86 runnable [0x00007f36daeb8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058a86a898> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
>> nid=0x3b85 runnable [0x00007f36d7e3b000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058abea788> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
>> tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
>> waiting on condition [0x00007f36d7b3a000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000058aecd008> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
>> runnable [0x00007f36db4be000]
>>    java.lang.Thread.State: RUNNABLE
>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> at
>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> at
>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
>> tid=0x00007f36f8025000 nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> at
>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
>> nid=0x3a96 runnable [0x00007f36dc0ea000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x0000000589202c58> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
>> nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>
>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
>> nid=0x3a6c runnable [0x00007f36dcb14000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058a825c18> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
>> tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005897334d0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
>> tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
>> at java.lang.Object.wait(Object.java:502)
>> at java.util.TimerThread.mainLoop(Timer.java:526)
>> - locked <0x00000005896b0d20> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
>> tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
>> at java.lang.Object.wait(Object.java:502)
>> at java.util.TimerThread.mainLoop(Timer.java:526)
>> - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
>> tid=0x00007f36f03a1000 nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>> at
>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
>> tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005895675a8> (a
>> scala.concurrent.forkjoin.ForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
>> tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
>> - locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> at
>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
>> nid=0x3a62 runnable [0x00007f36da5b1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
>> nid=0x3a61 runnable [0x00007f36da6b2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
>> - locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
>> nid=0x3a60 runnable [0x00007f36da7b3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
>> - locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
>> nid=0x3a5f runnable [0x00007f36da8b4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
>> nid=0x3a5e runnable [0x00007f36da9b5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
>> - locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
>> os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
>> [0x00007f36dacb6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000589567648> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
>> os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
>> [0x00007f36dadb7000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000589567648> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>> tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> at
>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
>> nid=0x3a43 runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
>> nid=0x3a42 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
>> nid=0x3a41 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
>> nid=0x3a40 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
>> nid=0x3a3f runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>> tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d
>> in Object.wait() [0x00007f36f51e5000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>
>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
>> nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>>
>> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
>> condition [0x00007f3712fda000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000058a974ab8> (a
>> java.util.concurrent.CountDownLatch$Sync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>> at
>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>> at
>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> at
>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> at scala.concurrent.Await$.ready(package.scala:86)
>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>> at
>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>> at
>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> at
>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>> at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>
>> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>>
>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
>> nid=0x3a34 runnable
>>
>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
>> nid=0x3a35 runnable
>>
>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
>> nid=0x3a36 runnable
>>
>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
>> nid=0x3a37 runnable
>>
>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
>> nid=0x3a3a runnable
>>
>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
>> waiting on condition
>>
>> JNI global references: 486
>>
>>
>>
>>
>>
>>
>>
>>
>> And the driver:
>>
>> 2015-03-27 02:20:47
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>
>> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
>> nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e67194f0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
>> nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e67194f0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
>> nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e67194f0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
>> nid=0x41d0 waiting on condition [0x00007fd38feef000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e67194f0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
>> tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000fd545eb8> (a
>> scala.concurrent.forkjoin.ForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5
>> os_prio=0 tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition
>> [0x00007fd3b8400000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e64f49f8> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
>> nid=0x417f waiting on condition [0x00007fd398935000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e67194f0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "task-result-getter-3" #107 daemon prio=5 os_prio=0
>> tid=0x00007fd3b003f800 nid=0x409c waiting on condition [0x00007fd3902f1000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e66e9a00> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "task-result-getter-2" #106 daemon prio=5 os_prio=0
>> tid=0x00007fd3b003d000 nid=0x409b waiting on condition [0x00007fd3903f2000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e66e9a00> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "task-result-getter-1" #105 daemon prio=5 os_prio=0
>> tid=0x00007fd3b003b800 nid=0x409a waiting on condition [0x00007fd3904f3000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e66e9a00> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "task-result-getter-0" #104 daemon prio=5 os_prio=0
>> tid=0x00007fd3b0030800 nid=0x4098 waiting on condition [0x00007fd3905f4000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e66e9a00> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
>> tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> at
>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
>> nid=0x4092 runnable [0x00007fd398b37000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e685f320> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
>> nid=0x4091 runnable [0x00007fd398a36000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e6871c30> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
>> nid=0x4090 runnable [0x00007fd39903a000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e6873520> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
>> tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
>> tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e7f19070> (a
>> scala.concurrent.forkjoin.ForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5
>> os_prio=0 tid=0x00007fd3a400d800 nid=0x4051 waiting on condition
>> [0x00007fd398e38000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e64f49f8> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
>> tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e66fe220> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
>> tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e740fa58> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
>> tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e740fce8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
>> tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
>> tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e737a0c8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
>> tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e737add8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
>> tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
>> tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
>> at
>> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>>
>> "ResponseProcessor for block
>> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
>> prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
>> [0x00007fd39a941000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
>> - locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>> at java.io.FilterInputStream.read(FilterInputStream.java:83)
>> at java.io.FilterInputStream.read(FilterInputStream.java:83)
>> at
>> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
>> at
>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>>
>> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
>> nid=0x3fda waiting on condition [0x00007fd39aa42000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e58145f0> (a
>> java.util.concurrent.Semaphore$NonfairSync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
>> at
>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
>> at
>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>> at
>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>> at
>> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>>
>> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0
>> tid=0x00007fd43258c000 nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
>> at
>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
>> at
>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>> at
>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>> at org.apache.spark.ContextCleaner.org
>> <http://org.apache.spark.contextcleaner.org/>
>> $apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
>> at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>>
>> "DataStreamer for file
>> /spark-logs/application_1427416187821_0004.inprogress block
>> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon
>> prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait()
>> [0x00007fd39ae44000]
>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
>> - locked <0x00000000e6994f30> (a java.util.LinkedList)
>>
>> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
>> tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e6995e88> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
>> nid=0x3fb7 runnable [0x00007fd39b246000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e6886b58> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>> - locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Yarn application state monitor" #73 daemon prio=5 os_prio=0
>> tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>>
>> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
>> nid=0x3f77 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f
>> waiting on condition [0x00007fd39b547000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e677ae98> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
>> tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
>> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e
>> runnable [0x00007fd3b80ff000]
>>    java.lang.Thread.State: RUNNABLE
>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> at
>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> at
>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop"
>> #62 daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in
>> Object.wait() [0x00007fd3b8501000]
>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
>> - locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>>
>> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
>> tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e66e9230> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
>> at
>> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>>
>> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f
>> in Object.wait() [0x00007fd3b8703000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at java.util.TimerThread.mainLoop(Timer.java:526)
>> - locked <0x00000000e66e8190> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
>> nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e67194f0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54
>> daemon prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
>> [0x00007fd3b8fb6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>> - locked <0x00000000e671a620> (a java.lang.Object)
>> at
>> org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>> at
>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
>> tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
>> - locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>> at
>> org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
>> tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at java.util.TimerThread.mainLoop(Timer.java:526)
>> - locked <0x00000000e66609f0> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
>> nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e6539dc8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
>> nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e6539dc8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
>> nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e6539dc8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
>> nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e6539dc8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
>> nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e6539dc8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
>> nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e6539dc8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
>> nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e6539dc8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon
>> prio=5 os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable
>> [0x00007fd3ba0de000]
>>    java.lang.Thread.State: RUNNABLE
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at
>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>> at java.net.ServerSocket.accept(ServerSocket.java:513)
>> at
>> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>> at
>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
>> tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at java.util.TimerThread.mainLoop(Timer.java:526)
>> - locked <0x00000000e6601270> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
>> tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at java.util.TimerThread.mainLoop(Timer.java:526)
>> - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
>> tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at java.util.TimerThread.mainLoop(Timer.java:526)
>> - locked <0x00000000e65380d8> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> "New I/O server boss #6" #36 daemon prio=5 os_prio=0
>> tid=0x00007fd39c04b800 nid=0x3df8 runnable [0x00007fd3ba6e2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
>> - locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> at
>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
>> nid=0x3df7 runnable [0x00007fd3ba7e3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
>> - locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
>> nid=0x3df6 runnable [0x00007fd3ba8e4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
>> - locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
>> nid=0x3df5 runnable [0x00007fd3ba9e5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
>> - locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
>> nid=0x3df4 runnable [0x00007fd3baae6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
>> - locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
>> nid=0x3df3 runnable [0x00007fd3babe7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
>> - locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
>> os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
>> [0x00007fd3baef9000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e64bbb40> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
>> os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
>> [0x00007fd3baffa000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e64bbb40> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0
>> tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e64f49f8> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0
>> tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e64f49f8> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0
>> tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e64f49f8> (a
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
>> tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> at
>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
>> nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e42808f8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
>> nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e42808f8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
>> nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e42808f8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
>> nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e42808f8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
>> nid=0x3d72 waiting on condition [0x00007fd40c460000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e42808f8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
>> nid=0x3d71 waiting on condition [0x00007fd40c561000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e42808f8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
>> nid=0x3d70 waiting on condition [0x00007fd40c662000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000000e42808f8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at
>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon
>> prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable
>> [0x00007fd40c966000]
>>    java.lang.Thread.State: RUNNABLE
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at
>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>> at java.net.ServerSocket.accept(ServerSocket.java:513)
>> at
>> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>> at
>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> at
>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
>> nid=0x3d67 runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
>> nid=0x3d66 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
>> nid=0x3d65 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
>> nid=0x3d64 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
>> nid=0x3d63 runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62
>> in Object.wait() [0x00007fd40f28e000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>
>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
>> nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>>
>> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
>> Object.wait() [0x00007fd438220000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
>> - locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
>> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
>> at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
>> at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
>> at
>> org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
>> at
>> org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
>> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
>> at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
>> at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
>> at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
>> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
>> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
>> at
>> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>> at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>> at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
>> at $line24.$read$$iwC$$iwC.<init>(<console>:35)
>> at $line24.$read$$iwC.<init>(<console>:37)
>> at $line24.$read.<init>(<console>:39)
>> at $line24.$read$.<init>(<console>:43)
>> at $line24.$read$.<clinit>(<console>)
>> at $line24.$eval$.<init>(<console>:7)
>> at $line24.$eval$.<clinit>(<console>)
>> at $line24.$eval.$print(<console>)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at
>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>> at
>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>> at
>> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
>> at
>> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
>> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
>> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
>> at org.apache.spark.repl.SparkILoop.org
>> <http://org.apache.spark.repl.sparkiloop.org/>
>> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>> at
>> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>> at org.apache.spark.repl.SparkILoop.org
>> <http://org.apache.spark.repl.sparkiloop.org/>
>> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
>> at org.apache.spark.repl.Main$.main(Main.scala:31)
>> at org.apache.spark.repl.Main.main(Main.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>>
>> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000
>> nid=0x3d5c runnable
>>
>> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800
>> nid=0x3d5d runnable
>>
>> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800
>> nid=0x3d5e runnable
>>
>> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000
>> nid=0x3d5f runnable
>>
>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
>> waiting on condition
>>
>> JNI global references: 420
>>
>>
>>
>> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>
>>> Is it possible to jstack the executors and see where they are hanging?
>>>
>>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:
>>>
>>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu,
>>>> 30GB), executor memory 20GB, driver memory 10GB
>>>>
>>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data
>>>> spread out over roughly 2,000 Parquet files and my queries frequently hang.
>>>> Simple queries like "select count(*) from ..." on the entire data set work
>>>> ok.  Slightly more demanding ones with group by's and some aggregate
>>>> functions (percentile_approx, avg, etc.) work ok as well, as long as I have
>>>> some criteria in my where clause to keep the number of rows down.
>>>>
>>>> Once I hit some limit on query complexity and rows processed, my
>>>> queries start to hang.  I've left them for up to an hour without seeing any
>>>> progress.  No OOM's either - the job is just stuck.
>>>>
>>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800,
>>>> but with the same results: usually near the end of the tasks (like 780 of
>>>> 800 complete), progress just stops:
>>>>
>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in
>>>> stage 1.0 (TID 1618) in 800 ms on ip-10-209-22-211.eu-west-1.compute.internal
>>>> (748/800)
>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in
>>>> stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>>> (749/800)
>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in
>>>> stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>>> (750/800)
>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in
>>>> stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>>> (751/800)
>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in
>>>> stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>>> (752/800)
>>>>
>>>> ^^^^^^^ this is where it stays forever
>>>>
>>>> Looking at the Spark UI, several of the executors still list active
>>>> tasks.  I do see that the Shuffle Read for executors that don't have any
>>>> tasks remaining is around 100MB, whereas it's more like 10MB for the
>>>> executors that still have tasks.
>>>>
>>>> The first stage, mapPartitions, always completes fine.  It's the second
>>>> stage (takeOrdered), that hangs.
>>>>
>>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
>>>> encountered it when using JSON files (instead of Parquet).
>>>>
>>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
>>>> are having this issue.
>>>>
>>>
>>>
>>
>>
>>
>

Re: Spark SQL queries hang forever

Posted by Jon Chase <jo...@gmail.com>.
Michael - nice find!

I can confirm that using nio for the block transfer service fixes the issue:

    --conf "spark.shuffle.blockTransferService=nio"

(note: setting this using sc.getConf(...) after the repl has started will
not work)

I've reported this as a bug:

https://issues.apache.org/jira/browse/SPARK-6962

Please vote for this in Jira if you're having the same issue.


On Thu, Apr 16, 2015 at 3:34 AM, Michael Allman <mi...@videoamp.com>
wrote:

> After many, many manual (and automated) trials using a spark sql query
> that *sometimes* triggers the hang I found the following:
>
>
>    - When I set spark.shuffle.blockTransferService to netty (the
>    default), I can sometimes reproduce the problem.
>    - When I set spark.shuffle.blockTransferService to nio (the Spark 1.1
>    default), I have not yet been able to reproduce the problem.
>
>
> While far from a reliable conclusion, the workaround I'm going with for
> now is setting "spark.shuffle.blockTransferService nio" in
> spark-defaults.conf. One of the reasons I homed in on this setting as a
> possible trigger was the fact that its default value changed starting in
> Spark 1.2. We haven't experienced a similar problem in our current Spark
> 1.1 cluster, and we're skipping directly to Spark 1.3. Has anyone else
> tried this apparent workaround?
>
> Cheers,
>
> Michael
>
> On Apr 15, 2015, at 11:23 PM, Michael Allman <mi...@videoamp.com> wrote:
>
> FWIW I'm attempting an upgrade from Spark 1.1 to Spark 1.3 and I'm seeing
> this exact problem in ShuffleBlockFetcherIterator in the query takeOrdered
> stage. Sometimes a single task hangs, sometimes a couple dozen. The final
> stage has 200 tasks. I'm using the Spark standalone scheduler, parquet
> files. Here's the stack trace of the stuck thread:
>
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:389)
>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:154)
>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:149)
> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> org.apache.spark.scheduler.Task.run(Task.scala:64)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
>
> Anyone find a reliable fix/workaround? I'm going to try some things
> myself. If I find something that works I'll post back.
>
> Cheers,
>
> Michael
>
>
> On Mar 26, 2015, at 7:21 PM, Jon Chase <jo...@gmail.com> wrote:
>
> Here are the jstacks from a 3 worker cluster.  Two of the workers are
> sitting with 4 active tasks, the other has no active tasks.  Not sure if it
> makes a difference, but I only see an ExecutorLauncher instance via JPS on
> the worker that has no active tasks.
>
> Here's the first stack dump, this machine has 4 active tasks:
>
> 2015-03-27 02:11:46
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
> nid=0x43a3 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
> os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
> [0x00007f91bad9a000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884fc0> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
> nid=0x42e2 runnable [0x00007f91b4c54000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005898aa000> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
> nid=0x42e1 runnable [0x00007f91b5d56000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058ad5b5f8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
> nid=0x42e0 runnable [0x00007f91b5e57000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005898aa0c0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
> nid=0x42df runnable [0x00007f91b5f58000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058ad5b6b8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cfda6800> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
> tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
> waiting on condition [0x00007f91b6c5b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058af8e0b8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
> runnable [0x00007f91baf9c000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0
> tid=0x00007f91d8118800 nid=0x41ee waiting on condition [0x00007f91bae9b000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
> nid=0x41ed runnable [0x00007f91baa97000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589224af0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
> tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006c8837750> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
> tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cc53bff0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
> tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cfda69b0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
> nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
> nid=0x41c6 runnable [0x00007f91bbac7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058984ee60> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
> tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005898f7290> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
> tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058974cb00> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
> tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000589904038> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x0000000589904038> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
> tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884e80> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800
> nid=0x41be waiting on condition [0x00007f91ba370000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O server boss #12" #39 daemon prio=5 os_prio=0
> tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
> - locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
> nid=0x41bc runnable [0x00007f91b9665000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
> - locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
> nid=0x41bb runnable [0x00007f91b9766000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
> - locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
> nid=0x41ba runnable [0x00007f91b9867000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
> nid=0x41b9 runnable [0x00007f91b9968000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
> - locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
> nid=0x41b8 runnable [0x00007f91b9c69000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
> - locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5
> os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
> [0x00007f91b9d6a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884f20> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5
> os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
> [0x00007f91b9e6b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884f20> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
> nid=0x419e runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
> nid=0x419d waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
> nid=0x419c waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
> nid=0x419b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
> nid=0x419a runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198
> in Object.wait() [0x00007f91d4254000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
> nid=0x4197 in Object.wait() [0x00007f91d4355000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
> condition [0x00007f91f1fdf000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589a1b5f0> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
> nid=0x4191 runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
> nid=0x4192 runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
> nid=0x4193 runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
> nid=0x4194 runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
> nid=0x4195 runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
> waiting on condition
>
> JNI global references: 319
>
>
>
>
>
>
>
>
>
>
>
>
> Here's the next worker, also for active tasks:
>
> 2015-03-27 02:12:54
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
> nid=0x44f1 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
> os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
> [0x00007fcbbf87f000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589510428> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
> tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006126cd358> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
> nid=0x4459 runnable [0x00007fcbc88c7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a6b8c98> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
> nid=0x4458 runnable [0x00007fcbc7897000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a6e5a18> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
> nid=0x4457 runnable [0x00007fcbc7998000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058aade110> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
> nid=0x4456 runnable [0x00007fcbc7b9a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058aade1d0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
> tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000615539dd8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
> tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000617bd4238> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006126cd538> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
> tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
> waiting on condition [0x00007fcbc4586000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058adabe70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
> runnable [0x00007fcbc7e9d000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> tid=0x00007fcbf9608800 nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
> nid=0x4364 runnable [0x00007fcbc8ac9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589116ae0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
> nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
> nid=0x4340 runnable [0x00007fcbc94f3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058964c068> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a6fa058> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058928c438> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058928c438> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058928c5a0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000
> nid=0x433a waiting on condition [0x00007fcbc85c4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a70a468> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
> - locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
> nid=0x4336 runnable [0x00007fcbc6f90000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
> - locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
> nid=0x4335 runnable [0x00007fcbc7091000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
> - locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
> nid=0x4334 runnable [0x00007fcbc7392000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
> - locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
> nid=0x4333 runnable [0x00007fcbc7493000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
> - locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
> nid=0x4332 runnable [0x00007fcbc7594000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
> - locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
> [0x00007fcbc7695000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005893c1820> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
> [0x00007fcbc7796000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005893c1820> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
> nid=0x4318 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
> nid=0x4317 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
> nid=0x4316 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
> nid=0x4315 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
> nid=0x4314 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312
> in Object.wait() [0x00007fcbe1be7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
> nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
> condition [0x00007fcbff9b9000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589585398> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
> nid=0x430b runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
> nid=0x430c runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
> nid=0x430d runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
> nid=0x430e runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
> nid=0x430f runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
> waiting on condition
>
> JNI global references: 375
>
>
>
>
>
>
>
>
>
>
> And the final worker, no active tasks:
>
>
> 2015-03-27 02:13:26
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
> nid=0x3c5b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
> os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
> [0x00007f36d2e33000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005895676e8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
> nid=0x3b88 runnable [0x00007f36dafb9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a86a7d8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
> nid=0x3b87 runnable [0x00007f36d7138000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058abea6c8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
> nid=0x3b86 runnable [0x00007f36daeb8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a86a898> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
> nid=0x3b85 runnable [0x00007f36d7e3b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058abea788> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
> tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
> waiting on condition [0x00007f36d7b3a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058aecd008> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
> runnable [0x00007f36db4be000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> tid=0x00007f36f8025000 nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
> nid=0x3a96 runnable [0x00007f36dc0ea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589202c58> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
> nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
> nid=0x3a6c runnable [0x00007f36dcb14000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a825c18> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005897334d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000005896b0d20> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000
> nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005895675a8> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
> - locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
> nid=0x3a62 runnable [0x00007f36da5b1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
> - locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
> nid=0x3a61 runnable [0x00007f36da6b2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
> - locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
> nid=0x3a60 runnable [0x00007f36da7b3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
> - locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
> nid=0x3a5f runnable [0x00007f36da8b4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
> - locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
> nid=0x3a5e runnable [0x00007f36da9b5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
> - locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
> [0x00007f36dacb6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589567648> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
> [0x00007f36dadb7000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589567648> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
> nid=0x3a43 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
> nid=0x3a42 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
> nid=0x3a41 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
> nid=0x3a40 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
> nid=0x3a3f runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d
> in Object.wait() [0x00007f36f51e5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
> nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
> condition [0x00007f3712fda000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a974ab8> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
> nid=0x3a34 runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
> nid=0x3a35 runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
> nid=0x3a36 runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
> nid=0x3a37 runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
> nid=0x3a3a runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
> waiting on condition
>
> JNI global references: 486
>
>
>
>
>
>
>
>
> And the driver:
>
> 2015-03-27 02:20:47
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
> nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
> nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
> nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
> nid=0x41d0 waiting on condition [0x00007fd38feef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
> tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000fd545eb8> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5
> os_prio=0 tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition
> [0x00007fd3b8400000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
> nid=0x417f waiting on condition [0x00007fd398935000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800
> nid=0x409c waiting on condition [0x00007fd3902f1000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000
> nid=0x409b waiting on condition [0x00007fd3903f2000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800
> nid=0x409a waiting on condition [0x00007fd3904f3000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800
> nid=0x4098 waiting on condition [0x00007fd3905f4000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
> tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
> nid=0x4092 runnable [0x00007fd398b37000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e685f320> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
> nid=0x4091 runnable [0x00007fd398a36000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6871c30> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
> nid=0x4090 runnable [0x00007fd39903a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6873520> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
> tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
> tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e7f19070> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0
> tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
> tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66fe220> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
> tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e740fa58> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
> tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e740fce8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
> tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
> tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e737a0c8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
> tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e737add8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
> tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> at java.lang.Thread.run(Thread.java:745)
>
> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
> tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
> at
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>
> "ResponseProcessor for block
> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
> prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
> [0x00007fd39a941000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
> - locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>
> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
> nid=0x3fda waiting on condition [0x00007fd39aa42000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e58145f0> (a
> java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>
> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000
> nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at org.apache.spark.ContextCleaner.org
> <http://org.apache.spark.contextcleaner.org/>
> $apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
> at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>
> "DataStreamer for file
> /spark-logs/application_1427416187821_0004.inprogress block
> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon
> prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait()
> [0x00007fd39ae44000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
> - locked <0x00000000e6994f30> (a java.util.LinkedList)
>
> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
> tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6995e88> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
> nid=0x3fb7 runnable [0x00007fd39b246000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6886b58> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Yarn application state monitor" #73 daemon prio=5 os_prio=0
> tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>
> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
> nid=0x3f77 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f
> waiting on condition [0x00007fd39b547000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e677ae98> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
> tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e
> runnable [0x00007fd3b80ff000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop"
> #62 daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in
> Object.wait() [0x00007fd3b8501000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
> - locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>
> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
> tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9230> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> at
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>
> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in
> Object.wait() [0x00007fd3b8703000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e66e8190> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
> nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54
> daemon prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
> [0x00007fd3b8fb6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> - locked <0x00000000e671a620> (a java.lang.Object)
> at
> org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
> tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> at
> org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
> tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e66609f0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
> nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
> nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
> nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
> nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
> nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
> nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
> nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon
> prio=5 os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable
> [0x00007fd3ba0de000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
> tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e6601270> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
> tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
> tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e65380d8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "New I/O server boss #6" #36 daemon prio=5 os_prio=0
> tid=0x00007fd39c04b800 nid=0x3df8 runnable [0x00007fd3ba6e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
> nid=0x3df7 runnable [0x00007fd3ba7e3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
> nid=0x3df6 runnable [0x00007fd3ba8e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
> nid=0x3df5 runnable [0x00007fd3ba9e5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
> - locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
> nid=0x3df4 runnable [0x00007fd3baae6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
> - locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
> nid=0x3df3 runnable [0x00007fd3babe7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
> - locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
> os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
> [0x00007fd3baef9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64bbb40> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
> os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
> [0x00007fd3baffa000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64bbb40> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0
> tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0
> tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0
> tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
> tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
> nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
> nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
> nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
> nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
> nid=0x3d72 waiting on condition [0x00007fd40c460000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
> nid=0x3d71 waiting on condition [0x00007fd40c561000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
> nid=0x3d70 waiting on condition [0x00007fd40c662000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon
> prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable
> [0x00007fd40c966000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
> nid=0x3d67 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
> nid=0x3d66 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
> nid=0x3d65 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
> nid=0x3d64 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
> nid=0x3d63 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62
> in Object.wait() [0x00007fd40f28e000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
> nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
> Object.wait() [0x00007fd438220000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> - locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
> at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
> at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
> at
> org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
> at
> org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
> at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
> at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
> at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
> at
> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
> at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
> at $line24.$read$$iwC$$iwC.<init>(<console>:35)
> at $line24.$read$$iwC.<init>(<console>:37)
> at $line24.$read.<init>(<console>:39)
> at $line24.$read$.<init>(<console>:43)
> at $line24.$read$.<clinit>(<console>)
> at $line24.$eval$.<init>(<console>:7)
> at $line24.$eval$.<clinit>(<console>)
> at $line24.$eval.$print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
> at org.apache.spark.repl.SparkILoop.org
> <http://org.apache.spark.repl.sparkiloop.org/>
> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.org
> <http://org.apache.spark.repl.sparkiloop.org/>
> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000
> nid=0x3d5c runnable
>
> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800
> nid=0x3d5d runnable
>
> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800
> nid=0x3d5e runnable
>
> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000
> nid=0x3d5f runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
> waiting on condition
>
> JNI global references: 420
>
>
>
> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> Is it possible to jstack the executors and see where they are hanging?
>>
>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:
>>
>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB),
>>> executor memory 20GB, driver memory 10GB
>>>
>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data
>>> spread out over roughly 2,000 Parquet files and my queries frequently hang.
>>> Simple queries like "select count(*) from ..." on the entire data set work
>>> ok.  Slightly more demanding ones with group by's and some aggregate
>>> functions (percentile_approx, avg, etc.) work ok as well, as long as I have
>>> some criteria in my where clause to keep the number of rows down.
>>>
>>> Once I hit some limit on query complexity and rows processed, my queries
>>> start to hang.  I've left them for up to an hour without seeing any
>>> progress.  No OOM's either - the job is just stuck.
>>>
>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but
>>> with the same results: usually near the end of the tasks (like 780 of 800
>>> complete), progress just stops:
>>>
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in
>>> stage 1.0 (TID 1618) in 800 ms on ip-10-209-22-211.eu-west-1.compute.internal
>>> (748/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in
>>> stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>> (749/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in
>>> stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>> (750/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in
>>> stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>> (751/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in
>>> stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>> (752/800)
>>>
>>> ^^^^^^^ this is where it stays forever
>>>
>>> Looking at the Spark UI, several of the executors still list active
>>> tasks.  I do see that the Shuffle Read for executors that don't have any
>>> tasks remaining is around 100MB, whereas it's more like 10MB for the
>>> executors that still have tasks.
>>>
>>> The first stage, mapPartitions, always completes fine.  It's the second
>>> stage (takeOrdered), that hangs.
>>>
>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
>>> encountered it when using JSON files (instead of Parquet).
>>>
>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
>>> are having this issue.
>>>
>>
>>
>
>
>

Re: Spark SQL queries hang forever

Posted by Reynold Xin <rx...@databricks.com>.
Thanks for the information.

Adding the dev list.

Do you mind trying 1.3.1 to see if the issue is fixed?  I could be wrong
but it migtht be related to https://issues.apache.org/jira/browse/SPARK-6578



On Thu, Apr 16, 2015 at 2:34 AM, Michael Allman <mi...@videoamp.com>
wrote:

> After many, many manual (and automated) trials using a spark sql query
> that *sometimes* triggers the hang I found the following:
>
>
>    - When I set spark.shuffle.blockTransferService to netty (the
>    default), I can sometimes reproduce the problem.
>    - When I set spark.shuffle.blockTransferService to nio (the Spark 1.1
>    default), I have not yet been able to reproduce the problem.
>
>
> While far from a reliable conclusion, the workaround I'm going with for
> now is setting "spark.shuffle.blockTransferService nio" in
> spark-defaults.conf. One of the reasons I homed in on this setting as a
> possible trigger was the fact that its default value changed starting in
> Spark 1.2. We haven't experienced a similar problem in our current Spark
> 1.1 cluster, and we're skipping directly to Spark 1.3. Has anyone else
> tried this apparent workaround?
>
> Cheers,
>
> Michael
>
> On Apr 15, 2015, at 11:23 PM, Michael Allman <mi...@videoamp.com> wrote:
>
> FWIW I'm attempting an upgrade from Spark 1.1 to Spark 1.3 and I'm seeing
> this exact problem in ShuffleBlockFetcherIterator in the query takeOrdered
> stage. Sometimes a single task hangs, sometimes a couple dozen. The final
> stage has 200 tasks. I'm using the Spark standalone scheduler, parquet
> files. Here's the stack trace of the stuck thread:
>
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:389)
>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:154)
>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:149)
> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> org.apache.spark.scheduler.Task.run(Task.scala:64)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
>
> Anyone find a reliable fix/workaround? I'm going to try some things
> myself. If I find something that works I'll post back.
>
> Cheers,
>
> Michael
>
>
> On Mar 26, 2015, at 7:21 PM, Jon Chase <jo...@gmail.com> wrote:
>
> Here are the jstacks from a 3 worker cluster.  Two of the workers are
> sitting with 4 active tasks, the other has no active tasks.  Not sure if it
> makes a difference, but I only see an ExecutorLauncher instance via JPS on
> the worker that has no active tasks.
>
> Here's the first stack dump, this machine has 4 active tasks:
>
> 2015-03-27 02:11:46
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
> nid=0x43a3 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
> os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
> [0x00007f91bad9a000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884fc0> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
> nid=0x42e2 runnable [0x00007f91b4c54000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005898aa000> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
> nid=0x42e1 runnable [0x00007f91b5d56000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058ad5b5f8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
> nid=0x42e0 runnable [0x00007f91b5e57000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005898aa0c0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
> nid=0x42df runnable [0x00007f91b5f58000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058ad5b6b8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cfda6800> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
> tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
> waiting on condition [0x00007f91b6c5b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058af8e0b8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
> runnable [0x00007f91baf9c000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0
> tid=0x00007f91d8118800 nid=0x41ee waiting on condition [0x00007f91bae9b000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
> nid=0x41ed runnable [0x00007f91baa97000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589224af0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
> tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006c8837750> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
> tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cc53bff0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
> tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cfda69b0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
> nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
> nid=0x41c6 runnable [0x00007f91bbac7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058984ee60> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
> tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005898f7290> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
> tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058974cb00> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
> tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000589904038> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x0000000589904038> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
> tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884e80> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800
> nid=0x41be waiting on condition [0x00007f91ba370000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O server boss #12" #39 daemon prio=5 os_prio=0
> tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
> - locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
> nid=0x41bc runnable [0x00007f91b9665000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
> - locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
> nid=0x41bb runnable [0x00007f91b9766000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
> - locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
> nid=0x41ba runnable [0x00007f91b9867000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
> nid=0x41b9 runnable [0x00007f91b9968000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
> - locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
> nid=0x41b8 runnable [0x00007f91b9c69000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
> - locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5
> os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
> [0x00007f91b9d6a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884f20> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5
> os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
> [0x00007f91b9e6b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884f20> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
> nid=0x419e runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
> nid=0x419d waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
> nid=0x419c waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
> nid=0x419b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
> nid=0x419a runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198
> in Object.wait() [0x00007f91d4254000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
> nid=0x4197 in Object.wait() [0x00007f91d4355000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
> condition [0x00007f91f1fdf000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589a1b5f0> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
> nid=0x4191 runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
> nid=0x4192 runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
> nid=0x4193 runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
> nid=0x4194 runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
> nid=0x4195 runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
> waiting on condition
>
> JNI global references: 319
>
>
>
>
>
>
>
>
>
>
>
>
> Here's the next worker, also for active tasks:
>
> 2015-03-27 02:12:54
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
> nid=0x44f1 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
> os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
> [0x00007fcbbf87f000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589510428> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
> tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006126cd358> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
> nid=0x4459 runnable [0x00007fcbc88c7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a6b8c98> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
> nid=0x4458 runnable [0x00007fcbc7897000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a6e5a18> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
> nid=0x4457 runnable [0x00007fcbc7998000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058aade110> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
> nid=0x4456 runnable [0x00007fcbc7b9a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058aade1d0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
> tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000615539dd8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
> tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000617bd4238> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006126cd538> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
> tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
> waiting on condition [0x00007fcbc4586000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058adabe70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
> runnable [0x00007fcbc7e9d000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> tid=0x00007fcbf9608800 nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
> nid=0x4364 runnable [0x00007fcbc8ac9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589116ae0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
> nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
> nid=0x4340 runnable [0x00007fcbc94f3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058964c068> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a6fa058> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058928c438> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058928c438> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058928c5a0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000
> nid=0x433a waiting on condition [0x00007fcbc85c4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a70a468> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
> - locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
> nid=0x4336 runnable [0x00007fcbc6f90000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
> - locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
> nid=0x4335 runnable [0x00007fcbc7091000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
> - locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
> nid=0x4334 runnable [0x00007fcbc7392000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
> - locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
> nid=0x4333 runnable [0x00007fcbc7493000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
> - locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
> nid=0x4332 runnable [0x00007fcbc7594000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
> - locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
> [0x00007fcbc7695000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005893c1820> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
> [0x00007fcbc7796000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005893c1820> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
> nid=0x4318 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
> nid=0x4317 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
> nid=0x4316 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
> nid=0x4315 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
> nid=0x4314 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312
> in Object.wait() [0x00007fcbe1be7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
> nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
> condition [0x00007fcbff9b9000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589585398> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
> nid=0x430b runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
> nid=0x430c runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
> nid=0x430d runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
> nid=0x430e runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
> nid=0x430f runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
> waiting on condition
>
> JNI global references: 375
>
>
>
>
>
>
>
>
>
>
> And the final worker, no active tasks:
>
>
> 2015-03-27 02:13:26
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
> nid=0x3c5b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
> os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
> [0x00007f36d2e33000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005895676e8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
> nid=0x3b88 runnable [0x00007f36dafb9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a86a7d8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
> nid=0x3b87 runnable [0x00007f36d7138000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058abea6c8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
> nid=0x3b86 runnable [0x00007f36daeb8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a86a898> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
> nid=0x3b85 runnable [0x00007f36d7e3b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058abea788> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
> tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
> waiting on condition [0x00007f36d7b3a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058aecd008> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
> runnable [0x00007f36db4be000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> tid=0x00007f36f8025000 nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
> nid=0x3a96 runnable [0x00007f36dc0ea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589202c58> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
> nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
> nid=0x3a6c runnable [0x00007f36dcb14000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a825c18> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005897334d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000005896b0d20> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000
> nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005895675a8> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
> - locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
> nid=0x3a62 runnable [0x00007f36da5b1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
> - locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
> nid=0x3a61 runnable [0x00007f36da6b2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
> - locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
> nid=0x3a60 runnable [0x00007f36da7b3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
> - locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
> nid=0x3a5f runnable [0x00007f36da8b4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
> - locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
> nid=0x3a5e runnable [0x00007f36da9b5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
> - locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
> [0x00007f36dacb6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589567648> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
> [0x00007f36dadb7000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589567648> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
> nid=0x3a43 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
> nid=0x3a42 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
> nid=0x3a41 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
> nid=0x3a40 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
> nid=0x3a3f runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d
> in Object.wait() [0x00007f36f51e5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
> nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
> condition [0x00007f3712fda000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a974ab8> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
> nid=0x3a34 runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
> nid=0x3a35 runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
> nid=0x3a36 runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
> nid=0x3a37 runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
> nid=0x3a3a runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
> waiting on condition
>
> JNI global references: 486
>
>
>
>
>
>
>
>
> And the driver:
>
> 2015-03-27 02:20:47
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
> nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
> nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
> nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
> nid=0x41d0 waiting on condition [0x00007fd38feef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
> tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000fd545eb8> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5
> os_prio=0 tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition
> [0x00007fd3b8400000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
> nid=0x417f waiting on condition [0x00007fd398935000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800
> nid=0x409c waiting on condition [0x00007fd3902f1000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000
> nid=0x409b waiting on condition [0x00007fd3903f2000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800
> nid=0x409a waiting on condition [0x00007fd3904f3000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800
> nid=0x4098 waiting on condition [0x00007fd3905f4000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
> tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
> nid=0x4092 runnable [0x00007fd398b37000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e685f320> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
> nid=0x4091 runnable [0x00007fd398a36000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6871c30> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
> nid=0x4090 runnable [0x00007fd39903a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6873520> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
> tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
> tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e7f19070> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0
> tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
> tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66fe220> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
> tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e740fa58> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
> tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e740fce8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
> tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
> tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e737a0c8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
> tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e737add8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
> tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> at java.lang.Thread.run(Thread.java:745)
>
> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
> tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
> at
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>
> "ResponseProcessor for block
> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
> prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
> [0x00007fd39a941000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
> - locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>
> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
> nid=0x3fda waiting on condition [0x00007fd39aa42000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e58145f0> (a
> java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>
> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000
> nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at org.apache.spark.ContextCleaner.org
> <http://org.apache.spark.contextcleaner.org/>
> $apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
> at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>
> "DataStreamer for file
> /spark-logs/application_1427416187821_0004.inprogress block
> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon
> prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait()
> [0x00007fd39ae44000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
> - locked <0x00000000e6994f30> (a java.util.LinkedList)
>
> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
> tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6995e88> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
> nid=0x3fb7 runnable [0x00007fd39b246000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6886b58> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Yarn application state monitor" #73 daemon prio=5 os_prio=0
> tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>
> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
> nid=0x3f77 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f
> waiting on condition [0x00007fd39b547000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e677ae98> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
> tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e
> runnable [0x00007fd3b80ff000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop"
> #62 daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in
> Object.wait() [0x00007fd3b8501000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
> - locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>
> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
> tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9230> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> at
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>
> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in
> Object.wait() [0x00007fd3b8703000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e66e8190> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
> nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54
> daemon prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
> [0x00007fd3b8fb6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> - locked <0x00000000e671a620> (a java.lang.Object)
> at
> org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
> tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> at
> org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
> tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e66609f0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
> nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
> nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
> nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
> nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
> nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
> nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
> nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon
> prio=5 os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable
> [0x00007fd3ba0de000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
> tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e6601270> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
> tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
> tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e65380d8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "New I/O server boss #6" #36 daemon prio=5 os_prio=0
> tid=0x00007fd39c04b800 nid=0x3df8 runnable [0x00007fd3ba6e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
> nid=0x3df7 runnable [0x00007fd3ba7e3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
> nid=0x3df6 runnable [0x00007fd3ba8e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
> nid=0x3df5 runnable [0x00007fd3ba9e5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
> - locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
> nid=0x3df4 runnable [0x00007fd3baae6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
> - locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
> nid=0x3df3 runnable [0x00007fd3babe7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
> - locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
> os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
> [0x00007fd3baef9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64bbb40> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
> os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
> [0x00007fd3baffa000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64bbb40> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0
> tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0
> tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0
> tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
> tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
> nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
> nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
> nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
> nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
> nid=0x3d72 waiting on condition [0x00007fd40c460000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
> nid=0x3d71 waiting on condition [0x00007fd40c561000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
> nid=0x3d70 waiting on condition [0x00007fd40c662000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon
> prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable
> [0x00007fd40c966000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
> nid=0x3d67 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
> nid=0x3d66 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
> nid=0x3d65 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
> nid=0x3d64 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
> nid=0x3d63 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62
> in Object.wait() [0x00007fd40f28e000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
> nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
> Object.wait() [0x00007fd438220000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> - locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
> at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
> at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
> at
> org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
> at
> org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
> at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
> at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
> at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
> at
> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
> at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
> at $line24.$read$$iwC$$iwC.<init>(<console>:35)
> at $line24.$read$$iwC.<init>(<console>:37)
> at $line24.$read.<init>(<console>:39)
> at $line24.$read$.<init>(<console>:43)
> at $line24.$read$.<clinit>(<console>)
> at $line24.$eval$.<init>(<console>:7)
> at $line24.$eval$.<clinit>(<console>)
> at $line24.$eval.$print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
> at org.apache.spark.repl.SparkILoop.org
> <http://org.apache.spark.repl.sparkiloop.org/>
> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.org
> <http://org.apache.spark.repl.sparkiloop.org/>
> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000
> nid=0x3d5c runnable
>
> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800
> nid=0x3d5d runnable
>
> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800
> nid=0x3d5e runnable
>
> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000
> nid=0x3d5f runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
> waiting on condition
>
> JNI global references: 420
>
>
>
> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> Is it possible to jstack the executors and see where they are hanging?
>>
>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:
>>
>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB),
>>> executor memory 20GB, driver memory 10GB
>>>
>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data
>>> spread out over roughly 2,000 Parquet files and my queries frequently hang.
>>> Simple queries like "select count(*) from ..." on the entire data set work
>>> ok.  Slightly more demanding ones with group by's and some aggregate
>>> functions (percentile_approx, avg, etc.) work ok as well, as long as I have
>>> some criteria in my where clause to keep the number of rows down.
>>>
>>> Once I hit some limit on query complexity and rows processed, my queries
>>> start to hang.  I've left them for up to an hour without seeing any
>>> progress.  No OOM's either - the job is just stuck.
>>>
>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but
>>> with the same results: usually near the end of the tasks (like 780 of 800
>>> complete), progress just stops:
>>>
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in
>>> stage 1.0 (TID 1618) in 800 ms on ip-10-209-22-211.eu-west-1.compute.internal
>>> (748/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in
>>> stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>> (749/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in
>>> stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>> (750/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in
>>> stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>> (751/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in
>>> stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>> (752/800)
>>>
>>> ^^^^^^^ this is where it stays forever
>>>
>>> Looking at the Spark UI, several of the executors still list active
>>> tasks.  I do see that the Shuffle Read for executors that don't have any
>>> tasks remaining is around 100MB, whereas it's more like 10MB for the
>>> executors that still have tasks.
>>>
>>> The first stage, mapPartitions, always completes fine.  It's the second
>>> stage (takeOrdered), that hangs.
>>>
>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
>>> encountered it when using JSON files (instead of Parquet).
>>>
>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
>>> are having this issue.
>>>
>>
>>
>
>
>

Re: Spark SQL queries hang forever

Posted by Michael Allman <mi...@videoamp.com>.
After many, many manual (and automated) trials using a spark sql query that *sometimes* triggers the hang I found the following:

When I set spark.shuffle.blockTransferService to netty (the default), I can sometimes reproduce the problem.
When I set spark.shuffle.blockTransferService to nio (the Spark 1.1 default), I have not yet been able to reproduce the problem.

While far from a reliable conclusion, the workaround I'm going with for now is setting "spark.shuffle.blockTransferService nio" in spark-defaults.conf. One of the reasons I homed in on this setting as a possible trigger was the fact that its default value changed starting in Spark 1.2. We haven't experienced a similar problem in our current Spark 1.1 cluster, and we're skipping directly to Spark 1.3. Has anyone else tried this apparent workaround?

Cheers,

Michael

> On Apr 15, 2015, at 11:23 PM, Michael Allman <mi...@videoamp.com> wrote:
> 
> FWIW I'm attempting an upgrade from Spark 1.1 to Spark 1.3 and I'm seeing this exact problem in ShuffleBlockFetcherIterator in the query takeOrdered stage. Sometimes a single task hangs, sometimes a couple dozen. The final stage has 200 tasks. I'm using the Spark standalone scheduler, parquet files. Here's the stack trace of the stuck thread:
> 
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:389)
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:154)
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:149)
> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> org.apache.spark.scheduler.Task.run(Task.scala:64)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
> 
> Anyone find a reliable fix/workaround? I'm going to try some things myself. If I find something that works I'll post back.
> 
> Cheers,
> 
> Michael
> 
> 
>> On Mar 26, 2015, at 7:21 PM, Jon Chase <jon.chase@gmail.com <ma...@gmail.com>> wrote:
>> 
>> Here are the jstacks from a 3 worker cluster.  Two of the workers are sitting with 4 active tasks, the other has no active tasks.  Not sure if it makes a difference, but I only see an ExecutorLauncher instance via JPS on the worker that has no active tasks.  
>> 
>> Here's the first stack dump, this machine has 4 active tasks:
>> 
>> 2015-03-27 02:11:46
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>> 
>> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000 nid=0x43a3 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5 os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition [0x00007f91bad9a000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x0000000589884fc0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000 nid=0x42e2 runnable [0x00007f91b4c54000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000005898aa000> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800 nid=0x42e1 runnable [0x00007f91b5d56000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058ad5b5f8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000 nid=0x42e0 runnable [0x00007f91b5e57000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000005898aa0c0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800 nid=0x42df runnable [0x00007f91b5f58000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058ad5b6b8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000006cfda6800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0 tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2 waiting on condition [0x00007f91b6c5b000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x000000058af8e0b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef runnable [0x00007f91baf9c000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0 tid=0x00007f91d8118800 nid=0x41ee waiting on condition [0x00007f91bae9b000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000 nid=0x41ed runnable [0x00007f91baa97000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x0000000589224af0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0 tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000006c8837750> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0 tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000006cc53bff0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0 tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000006cfda69b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800 nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>> 
>> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800 nid=0x41c6 runnable [0x00007f91bbac7000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058984ee60> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0 tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000005898f7290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0 tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	- waiting on <0x000000058974cb00> (a java.util.TaskQueue)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>> 	- locked <0x000000058974cb00> (a java.util.TaskQueue)
>> 	at java.util.TimerThread.run(Timer.java:505)
>> 
>> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0 tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	- waiting on <0x0000000589904038> (a java.util.TaskQueue)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>> 	- locked <0x0000000589904038> (a java.util.TaskQueue)
>> 	at java.util.TimerThread.run(Timer.java:505)
>> 
>> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0 tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x0000000589884e80> (a scala.concurrent.forkjoin.ForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800 nid=0x41be waiting on condition [0x00007f91ba370000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O server boss #12" #39 daemon prio=5 os_prio=0 tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058992f188> (a sun.nio.ch.Util$2)
>> 	- locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000 nid=0x41bc runnable [0x00007f91b9665000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x0000000589931218> (a sun.nio.ch.Util$2)
>> 	- locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800 nid=0x41bb runnable [0x00007f91b9766000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
>> 	- locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800 nid=0x41ba runnable [0x00007f91b9867000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
>> 	- locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800 nid=0x41b9 runnable [0x00007f91b9968000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
>> 	- locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000 nid=0x41b8 runnable [0x00007f91b9c69000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
>> 	- locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5 os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition [0x00007f91b9d6a000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5 os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition [0x00007f91b9e6b000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800 nid=0x419e runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000 nid=0x419d waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000 nid=0x419c waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000 nid=0x419b waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800 nid=0x419a runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198 in Object.wait() [0x00007f91d4254000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> 	- locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>> 
>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800 nid=0x4197 in Object.wait() [0x00007f91d4355000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> 	- locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>> 
>> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on condition [0x00007f91f1fdf000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x0000000589a1b5f0> (a java.util.concurrent.CountDownLatch$Sync)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> 	at scala.concurrent.Await$.ready(package.scala:86)
>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>> 
>> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>> 
>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000 nid=0x4191 runnable
>> 
>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800 nid=0x4192 runnable
>> 
>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800 nid=0x4193 runnable
>> 
>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000 nid=0x4194 runnable
>> 
>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000 nid=0x4195 runnable
>> 
>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f waiting on condition
>> 
>> JNI global references: 319
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Here's the next worker, also for active tasks:
>> 
>> 2015-03-27 02:12:54
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>> 
>> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000 nid=0x44f1 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5 os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition [0x00007fcbbf87f000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x0000000589510428> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0 tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000006126cd358> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800 nid=0x4459 runnable [0x00007fcbc88c7000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058a6b8c98> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000 nid=0x4458 runnable [0x00007fcbc7897000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058a6e5a18> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800 nid=0x4457 runnable [0x00007fcbc7998000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058aade110> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000 nid=0x4456 runnable [0x00007fcbc7b9a000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058aade1d0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0 tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x0000000615539dd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0 tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x0000000617bd4238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000006126cd538> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0 tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369 waiting on condition [0x00007fcbc4586000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x000000058adabe70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366 runnable [0x00007fcbc7e9d000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007fcbf9608800 nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800 nid=0x4364 runnable [0x00007fcbc8ac9000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x0000000589116ae0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000 nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>> 
>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000 nid=0x4340 runnable [0x00007fcbc94f3000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058964c068> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x000000058a6fa058> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	- waiting on <0x000000058928c438> (a java.util.TaskQueue)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>> 	- locked <0x000000058928c438> (a java.util.TaskQueue)
>> 	at java.util.TimerThread.run(Timer.java:505)
>> 
>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	- waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>> 	- locked <0x000000058928c5a0> (a java.util.TaskQueue)
>> 	at java.util.TimerThread.run(Timer.java:505)
>> 
>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000 nid=0x433a waiting on condition [0x00007fcbc85c4000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x000000058a70a468> (a scala.concurrent.forkjoin.ForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
>> 	- locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000 nid=0x4336 runnable [0x00007fcbc6f90000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
>> 	- locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800 nid=0x4335 runnable [0x00007fcbc7091000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
>> 	- locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000 nid=0x4334 runnable [0x00007fcbc7392000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058a796550> (a sun.nio.ch.Util$2)
>> 	- locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000 nid=0x4333 runnable [0x00007fcbc7493000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
>> 	- locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000 nid=0x4332 runnable [0x00007fcbc7594000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
>> 	- locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition [0x00007fcbc7695000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition [0x00007fcbc7796000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800 nid=0x4318 runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800 nid=0x4317 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000 nid=0x4316 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000 nid=0x4315 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000 nid=0x4314 runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312 in Object.wait() [0x00007fcbe1be7000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> 	- locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>> 
>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800 nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> 	- locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>> 
>> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on condition [0x00007fcbff9b9000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x0000000589585398> (a java.util.concurrent.CountDownLatch$Sync)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> 	at scala.concurrent.Await$.ready(package.scala:86)
>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>> 
>> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>> 
>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000 nid=0x430b runnable
>> 
>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800 nid=0x430c runnable
>> 
>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800 nid=0x430d runnable
>> 
>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000 nid=0x430e runnable
>> 
>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000 nid=0x430f runnable
>> 
>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319 waiting on condition
>> 
>> JNI global references: 375
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> And the final worker, no active tasks:
>> 
>> 
>> 2015-03-27 02:13:26
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>> 
>> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000 nid=0x3c5b waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5 os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition [0x00007f36d2e33000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000005895676e8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000 nid=0x3b88 runnable [0x00007f36dafb9000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058a86a7d8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800 nid=0x3b87 runnable [0x00007f36d7138000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058abea6c8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800 nid=0x3b86 runnable [0x00007f36daeb8000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058a86a898> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000 nid=0x3b85 runnable [0x00007f36d7e3b000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058abea788> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0 tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d waiting on condition [0x00007f36d7b3a000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x000000058aecd008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a runnable [0x00007f36db4be000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007f36f8025000 nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000 nid=0x3a96 runnable [0x00007f36dc0ea000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x0000000589202c58> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000 nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>> 
>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000 nid=0x3a6c runnable [0x00007f36dcb14000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058a825c18> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000005897334d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	- waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>> 	- locked <0x00000005896b0d20> (a java.util.TaskQueue)
>> 	at java.util.TimerThread.run(Timer.java:505)
>> 
>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	- waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>> 	- locked <0x00000005896b0dc8> (a java.util.TaskQueue)
>> 	at java.util.TimerThread.run(Timer.java:505)
>> 
>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000 nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000005895675a8> (a scala.concurrent.forkjoin.ForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058970f538> (a sun.nio.ch.Util$2)
>> 	- locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800 nid=0x3a62 runnable [0x00007f36da5b1000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800 nid=0x3a61 runnable [0x00007f36da6b2000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000 nid=0x3a60 runnable [0x00007f36da7b3000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
>> 	- locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800 nid=0x3a5f runnable [0x00007f36da8b4000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800 nid=0x3a5e runnable [0x00007f36da9b5000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition [0x00007f36dacb6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition [0x00007f36dadb7000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800 nid=0x3a43 runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000 nid=0x3a42 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000 nid=0x3a41 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000 nid=0x3a40 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800 nid=0x3a3f runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d in Object.wait() [0x00007f36f51e5000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> 	- locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>> 
>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800 nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> 	- locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>> 
>> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on condition [0x00007f3712fda000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x000000058a974ab8> (a java.util.concurrent.CountDownLatch$Sync)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> 	at scala.concurrent.Await$.ready(package.scala:86)
>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>> 
>> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>> 
>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000 nid=0x3a34 runnable
>> 
>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800 nid=0x3a35 runnable
>> 
>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800 nid=0x3a36 runnable
>> 
>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000 nid=0x3a37 runnable
>> 
>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000 nid=0x3a3a runnable
>> 
>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44 waiting on condition
>> 
>> JNI global references: 486
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> And the driver:
>> 
>> 2015-03-27 02:20:47
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>> 
>> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800 nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000 nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800 nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000 nid=0x41d0 waiting on condition [0x00007fd38feef000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0 tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000fd545eb8> (a scala.concurrent.forkjoin.ForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5 os_prio=0 tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition [0x00007fd3b8400000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000 nid=0x417f waiting on condition [0x00007fd398935000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800 nid=0x409c waiting on condition [0x00007fd3902f1000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000 nid=0x409b waiting on condition [0x00007fd3903f2000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800 nid=0x409a waiting on condition [0x00007fd3904f3000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800 nid=0x4098 waiting on condition [0x00007fd3905f4000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0 tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800 nid=0x4092 runnable [0x00007fd398b37000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e685f320> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800 nid=0x4091 runnable [0x00007fd398a36000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e6871c30> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800 nid=0x4090 runnable [0x00007fd39903a000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e6873520> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0 tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0 tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e7f19070> (a scala.concurrent.forkjoin.ForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0 tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0 tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e66fe220> (a java.util.concurrent.SynchronousQueue$TransferStack)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> 	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> 	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0 tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e740fa58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0 tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e740fce8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0 tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> 	- locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0 tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e737a0c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0 tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e737add8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0 tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> 	- locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0 tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> 	- locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
>> 	at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>> 
>> "ResponseProcessor for block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable [0x00007fd39a941000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
>> 	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
>> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
>> 	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
>> 	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
>> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>> 
>> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000 nid=0x3fda waiting on condition [0x00007fd39aa42000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e58145f0> (a java.util.concurrent.Semaphore$NonfairSync)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> 	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>> 
>> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000 nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> 	- locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>> 	at org.apache.spark.ContextCleaner.org <http://org.apache.spark.contextcleaner.org/>$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
>> 	at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>> 
>> "DataStreamer for file /spark-logs/application_1427416187821_0004.inprogress block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait() [0x00007fd39ae44000]
>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
>> 	- locked <0x00000000e6994f30> (a java.util.LinkedList)
>> 
>> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0 tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e6995e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800 nid=0x3fb7 runnable [0x00007fd39b246000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e6886b58> (a io.netty.channel.nio.SelectedSelectionKeySet)
>> 	- locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Yarn application state monitor" #73 daemon prio=5 os_prio=0 tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>> 
>> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000 nid=0x3f77 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f waiting on condition [0x00007fd39b547000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e677ae98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "LeaseRenewer:hadoop@10.209.23.74:9000 <http://LeaseRenewer:hadoop@10.209.23.74:9000/>" #68 daemon prio=5 os_prio=0 tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
>> 	at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>> 	at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e runnable [0x00007fd3b80ff000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "IPC Client (1489870911) connection to /10.209.23.74:9022 <http://10.209.23.74:9022/> from hadoop" #62 daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in Object.wait() [0x00007fd3b8501000]
>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
>> 	- locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
>> 	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>> 
>> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0 tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e66e9230> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> 	at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
>> 	at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
>> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>> 
>> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in Object.wait() [0x00007fd3b8703000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>> 	- locked <0x00000000e66e8190> (a java.util.TaskQueue)
>> 	at java.util.TimerThread.run(Timer.java:505)
>> 
>> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000 nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040 <http://SelectChannelConnector@0.0.0.0:4040/>" #54 daemon prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable [0x00007fd3b8fb6000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>> 	- locked <0x00000000e671a620> (a java.lang.Object)
>> 	at org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0 tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>> 	at org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0 tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>> 	- locked <0x00000000e66609f0> (a java.util.TaskQueue)
>> 	at java.util.TimerThread.run(Timer.java:505)
>> 
>> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000 nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800 nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800 nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800 nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800 nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800 nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000 nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920 <http://SocketConnector@0.0.0.0:53920/>" #42 daemon prio=5 os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable [0x00007fd3ba0de000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
>> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
>> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0 tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>> 	- locked <0x00000000e6601270> (a java.util.TaskQueue)
>> 	at java.util.TimerThread.run(Timer.java:505)
>> 
>> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0 tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>> 	- locked <0x00000000e6600ca8> (a java.util.TaskQueue)
>> 	at java.util.TimerThread.run(Timer.java:505)
>> 
>> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0 tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>> 	- locked <0x00000000e65380d8> (a java.util.TaskQueue)
>> 	at java.util.TimerThread.run(Timer.java:505)
>> 
>> "New I/O server boss #6" #36 daemon prio=5 os_prio=0 tid=0x00007fd39c04b800 nid=0x3df8 runnable [0x00007fd3ba6e2000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800 nid=0x3df7 runnable [0x00007fd3ba7e3000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800 nid=0x3df6 runnable [0x00007fd3ba8e4000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000 nid=0x3df5 runnable [0x00007fd3ba9e5000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800 nid=0x3df4 runnable [0x00007fd3baae6000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000 nid=0x3df3 runnable [0x00007fd3babe7000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>> 	- locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
>> 	- locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
>> 	- locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5 os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition [0x00007fd3baef9000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5 os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition [0x00007fd3baffa000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0 tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0 tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0 tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>>    java.lang.Thread.State: WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 
>> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0 tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>> 	at java.lang.Thread.sleep(Native Method)
>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800 nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000 nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800 nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800 nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000 nid=0x3d72 waiting on condition [0x00007fd40c460000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000 nid=0x3d71 waiting on condition [0x00007fd40c561000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000 nid=0x3d70 waiting on condition [0x00007fd40c662000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> 	at sun.misc.Unsafe.park(Native Method)
>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131 <http://SocketConnector@0.0.0.0:53131/>" #14 daemon prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable [0x00007fd40c966000]
>>    java.lang.Thread.State: RUNNABLE
>> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
>> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
>> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000 nid=0x3d67 runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800 nid=0x3d66 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000 nid=0x3d65 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000 nid=0x3d64 waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800 nid=0x3d63 runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>> 
>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62 in Object.wait() [0x00007fd40f28e000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>> 	- locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>> 
>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800 nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>> 	- locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>> 
>> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in Object.wait() [0x00007fd438220000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> 	at java.lang.Object.wait(Native Method)
>> 	at java.lang.Object.wait(Object.java:502)
>> 	at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
>> 	- locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
>> 	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
>> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
>> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
>> 	at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
>> 	at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
>> 	at org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
>> 	at org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
>> 	at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
>> 	at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
>> 	at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
>> 	at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
>> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
>> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>> 	at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
>> 	at $line24.$read$$iwC$$iwC.<init>(<console>:35)
>> 	at $line24.$read$$iwC.<init>(<console>:37)
>> 	at $line24.$read.<init>(<console>:39)
>> 	at $line24.$read$.<init>(<console>:43)
>> 	at $line24.$read$.<clinit>(<console>)
>> 	at $line24.$eval$.<init>(<console>:7)
>> 	at $line24.$eval$.<clinit>(<console>)
>> 	at $line24.$eval.$print(<console>)
>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> 	at java.lang.reflect.Method.invoke(Method.java:497)
>> 	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>> 	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>> 	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>> 	at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
>> 	at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
>> 	at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
>> 	at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
>> 	at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
>> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>> 	at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>> 	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
>> 	at org.apache.spark.repl.Main$.main(Main.scala:31)
>> 	at org.apache.spark.repl.Main.main(Main.scala)
>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> 	at java.lang.reflect.Method.invoke(Method.java:497)
>> 	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>> 	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>> 	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>> 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>> 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> 
>> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>> 
>> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000 nid=0x3d5c runnable
>> 
>> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800 nid=0x3d5d runnable
>> 
>> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800 nid=0x3d5e runnable
>> 
>> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000 nid=0x3d5f runnable
>> 
>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68 waiting on condition
>> 
>> JNI global references: 420
>> 
>> 
>> 
>> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <michael@databricks.com <ma...@databricks.com>> wrote:
>> Is it possible to jstack the executors and see where they are hanging?
>> 
>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jon.chase@gmail.com <ma...@gmail.com>> wrote:
>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB), executor memory 20GB, driver memory 10GB
>> 
>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data spread out over roughly 2,000 Parquet files and my queries frequently hang. Simple queries like "select count(*) from ..." on the entire data set work ok.  Slightly more demanding ones with group by's and some aggregate functions (percentile_approx, avg, etc.) work ok as well, as long as I have some criteria in my where clause to keep the number of rows down.  
>> 
>> Once I hit some limit on query complexity and rows processed, my queries start to hang.  I've left them for up to an hour without seeing any progress.  No OOM's either - the job is just stuck.
>> 
>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but with the same results: usually near the end of the tasks (like 780 of 800 complete), progress just stops:
>> 
>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in stage 1.0 (TID 1618) in 800 ms on ip-10-209-22-211.eu <http://ip-10-209-22-211.eu/>-west-1.compute.internal (748/800)
>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu <http://ip-10-105-12-41.eu/>-west-1.compute.internal (749/800)
>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu <http://ip-10-90-2-201.eu/>-west-1.compute.internal (750/800)
>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu <http://ip-10-90-2-201.eu/>-west-1.compute.internal (751/800)
>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu <http://ip-10-105-12-41.eu/>-west-1.compute.internal (752/800)
>> 
>> ^^^^^^^ this is where it stays forever
>> 
>> Looking at the Spark UI, several of the executors still list active tasks.  I do see that the Shuffle Read for executors that don't have any tasks remaining is around 100MB, whereas it's more like 10MB for the executors that still have tasks.  
>> 
>> The first stage, mapPartitions, always completes fine.  It's the second stage (takeOrdered), that hangs.
>> 
>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also encountered it when using JSON files (instead of Parquet).
>> 
>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do are having this issue.
>> 
>> 
> 


Re: Spark SQL queries hang forever

Posted by Michael Allman <mi...@videoamp.com>.
FWIW I'm attempting an upgrade from Spark 1.1 to Spark 1.3 and I'm seeing this exact problem in ShuffleBlockFetcherIterator in the query takeOrdered stage. Sometimes a single task hangs, sometimes a couple dozen. The final stage has 200 tasks. I'm using the Spark standalone scheduler, parquet files. Here's the stack trace of the stuck thread:

sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:389)
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:154)
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:149)
org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
org.apache.spark.scheduler.Task.run(Task.scala:64)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)

Anyone find a reliable fix/workaround? I'm going to try some things myself. If I find something that works I'll post back.

Cheers,

Michael


> On Mar 26, 2015, at 7:21 PM, Jon Chase <jo...@gmail.com> wrote:
> 
> Here are the jstacks from a 3 worker cluster.  Two of the workers are sitting with 4 active tasks, the other has no active tasks.  Not sure if it makes a difference, but I only see an ExecutorLauncher instance via JPS on the worker that has no active tasks.  
> 
> Here's the first stack dump, this machine has 4 active tasks:
> 
> 2015-03-27 02:11:46
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
> 
> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000 nid=0x43a3 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5 os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition [0x00007f91bad9a000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000589884fc0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000 nid=0x42e2 runnable [0x00007f91b4c54000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000005898aa000> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800 nid=0x42e1 runnable [0x00007f91b5d56000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058ad5b5f8> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000 nid=0x42e0 runnable [0x00007f91b5e57000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000005898aa0c0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800 nid=0x42df runnable [0x00007f91b5f58000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058ad5b6b8> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000006cfda6800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0 tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2 waiting on condition [0x00007f91b6c5b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x000000058af8e0b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef runnable [0x00007f91baf9c000]
>    java.lang.Thread.State: RUNNABLE
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0 tid=0x00007f91d8118800 nid=0x41ee waiting on condition [0x00007f91bae9b000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000 nid=0x41ed runnable [0x00007f91baa97000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x0000000589224af0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0 tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000006c8837750> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0 tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000006cc53bff0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0 tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000006cfda69b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800 nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
> 
> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800 nid=0x41c6 runnable [0x00007f91bbac7000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058984ee60> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0 tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000005898f7290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0 tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x000000058974cb00> (a java.util.TaskQueue)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x000000058974cb00> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> 
> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0 tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x0000000589904038> (a java.util.TaskQueue)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x0000000589904038> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> 
> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0 tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000589884e80> (a scala.concurrent.forkjoin.ForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800 nid=0x41be waiting on condition [0x00007f91ba370000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O server boss #12" #39 daemon prio=5 os_prio=0 tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058992f188> (a sun.nio.ch.Util$2)
> 	- locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000 nid=0x41bc runnable [0x00007f91b9665000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x0000000589931218> (a sun.nio.ch.Util$2)
> 	- locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800 nid=0x41bb runnable [0x00007f91b9766000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
> 	- locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800 nid=0x41ba runnable [0x00007f91b9867000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
> 	- locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800 nid=0x41b9 runnable [0x00007f91b9968000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
> 	- locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000 nid=0x41b8 runnable [0x00007f91b9c69000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
> 	- locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5 os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition [0x00007f91b9d6a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5 os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition [0x00007f91b9e6b000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800 nid=0x419e runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000 nid=0x419d waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000 nid=0x419c waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000 nid=0x419b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800 nid=0x419a runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198 in Object.wait() [0x00007f91d4254000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> 	- locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> 
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800 nid=0x4197 in Object.wait() [0x00007f91d4355000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> 	- locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
> 
> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on condition [0x00007f91f1fdf000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000589a1b5f0> (a java.util.concurrent.CountDownLatch$Sync)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> 	at scala.concurrent.Await$.ready(package.scala:86)
> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> 
> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
> 
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000 nid=0x4191 runnable
> 
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800 nid=0x4192 runnable
> 
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800 nid=0x4193 runnable
> 
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000 nid=0x4194 runnable
> 
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000 nid=0x4195 runnable
> 
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f waiting on condition
> 
> JNI global references: 319
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Here's the next worker, also for active tasks:
> 
> 2015-03-27 02:12:54
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
> 
> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000 nid=0x44f1 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5 os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition [0x00007fcbbf87f000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000589510428> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0 tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000006126cd358> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800 nid=0x4459 runnable [0x00007fcbc88c7000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058a6b8c98> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000 nid=0x4458 runnable [0x00007fcbc7897000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058a6e5a18> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800 nid=0x4457 runnable [0x00007fcbc7998000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058aade110> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000 nid=0x4456 runnable [0x00007fcbc7b9a000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058aade1d0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0 tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000615539dd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0 tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000617bd4238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000006126cd538> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0 tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369 waiting on condition [0x00007fcbc4586000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x000000058adabe70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366 runnable [0x00007fcbc7e9d000]
>    java.lang.Thread.State: RUNNABLE
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007fcbf9608800 nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800 nid=0x4364 runnable [0x00007fcbc8ac9000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x0000000589116ae0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000 nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
> 
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000 nid=0x4340 runnable [0x00007fcbc94f3000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058964c068> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x000000058a6fa058> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x000000058928c438> (a java.util.TaskQueue)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x000000058928c438> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> 
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x000000058928c5a0> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> 
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000 nid=0x433a waiting on condition [0x00007fcbc85c4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x000000058a70a468> (a scala.concurrent.forkjoin.ForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
> 	- locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000 nid=0x4336 runnable [0x00007fcbc6f90000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
> 	- locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800 nid=0x4335 runnable [0x00007fcbc7091000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
> 	- locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000 nid=0x4334 runnable [0x00007fcbc7392000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058a796550> (a sun.nio.ch.Util$2)
> 	- locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000 nid=0x4333 runnable [0x00007fcbc7493000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
> 	- locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000 nid=0x4332 runnable [0x00007fcbc7594000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
> 	- locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition [0x00007fcbc7695000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition [0x00007fcbc7796000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800 nid=0x4318 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800 nid=0x4317 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000 nid=0x4316 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000 nid=0x4315 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000 nid=0x4314 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312 in Object.wait() [0x00007fcbe1be7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> 	- locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> 
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800 nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> 	- locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
> 
> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on condition [0x00007fcbff9b9000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000589585398> (a java.util.concurrent.CountDownLatch$Sync)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> 	at scala.concurrent.Await$.ready(package.scala:86)
> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> 
> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
> 
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000 nid=0x430b runnable
> 
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800 nid=0x430c runnable
> 
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800 nid=0x430d runnable
> 
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000 nid=0x430e runnable
> 
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000 nid=0x430f runnable
> 
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319 waiting on condition
> 
> JNI global references: 375
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> And the final worker, no active tasks:
> 
> 
> 2015-03-27 02:13:26
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
> 
> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000 nid=0x3c5b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5 os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition [0x00007f36d2e33000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000005895676e8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000 nid=0x3b88 runnable [0x00007f36dafb9000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058a86a7d8> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800 nid=0x3b87 runnable [0x00007f36d7138000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058abea6c8> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800 nid=0x3b86 runnable [0x00007f36daeb8000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058a86a898> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000 nid=0x3b85 runnable [0x00007f36d7e3b000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058abea788> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0 tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d waiting on condition [0x00007f36d7b3a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x000000058aecd008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a runnable [0x00007f36db4be000]
>    java.lang.Thread.State: RUNNABLE
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007f36f8025000 nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000 nid=0x3a96 runnable [0x00007f36dc0ea000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x0000000589202c58> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000 nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
> 
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000 nid=0x3a6c runnable [0x00007f36dcb14000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058a825c18> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000005897334d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x00000005896b0d20> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> 
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x00000005896b0dc8> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> 
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000 nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000005895675a8> (a scala.concurrent.forkjoin.ForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058970f538> (a sun.nio.ch.Util$2)
> 	- locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800 nid=0x3a62 runnable [0x00007f36da5b1000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
> 	- locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800 nid=0x3a61 runnable [0x00007f36da6b2000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
> 	- locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000 nid=0x3a60 runnable [0x00007f36da7b3000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
> 	- locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800 nid=0x3a5f runnable [0x00007f36da8b4000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
> 	- locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800 nid=0x3a5e runnable [0x00007f36da9b5000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
> 	- locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition [0x00007f36dacb6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition [0x00007f36dadb7000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800 nid=0x3a43 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000 nid=0x3a42 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000 nid=0x3a41 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000 nid=0x3a40 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800 nid=0x3a3f runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d in Object.wait() [0x00007f36f51e5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> 	- locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> 
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800 nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> 	- locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
> 
> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on condition [0x00007f3712fda000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x000000058a974ab8> (a java.util.concurrent.CountDownLatch$Sync)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> 	at scala.concurrent.Await$.ready(package.scala:86)
> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> 
> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
> 
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000 nid=0x3a34 runnable
> 
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800 nid=0x3a35 runnable
> 
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800 nid=0x3a36 runnable
> 
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000 nid=0x3a37 runnable
> 
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000 nid=0x3a3a runnable
> 
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44 waiting on condition
> 
> JNI global references: 486
> 
> 
> 
> 
> 
> 
> 
> 
> And the driver:
> 
> 2015-03-27 02:20:47
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
> 
> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800 nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000 nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800 nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000 nid=0x41d0 waiting on condition [0x00007fd38feef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0 tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000fd545eb8> (a scala.concurrent.forkjoin.ForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5 os_prio=0 tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition [0x00007fd3b8400000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000 nid=0x417f waiting on condition [0x00007fd398935000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800 nid=0x409c waiting on condition [0x00007fd3902f1000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000 nid=0x409b waiting on condition [0x00007fd3903f2000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800 nid=0x409a waiting on condition [0x00007fd3904f3000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800 nid=0x4098 waiting on condition [0x00007fd3905f4000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0 tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800 nid=0x4092 runnable [0x00007fd398b37000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e685f320> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800 nid=0x4091 runnable [0x00007fd398a36000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e6871c30> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800 nid=0x4090 runnable [0x00007fd39903a000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e6873520> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0 tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0 tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e7f19070> (a scala.concurrent.forkjoin.ForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0 tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0 tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e66fe220> (a java.util.concurrent.SynchronousQueue$TransferStack)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> 	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> 	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0 tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e740fa58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0 tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e740fce8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0 tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> 	- locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0 tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e737a0c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0 tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e737add8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0 tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> 	- locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0 tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> 	- locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
> 	at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
> 
> "ResponseProcessor for block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable [0x00007fd39a941000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
> 	- locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
> 	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
> 	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
> 	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
> 
> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000 nid=0x3fda waiting on condition [0x00007fd39aa42000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e58145f0> (a java.util.concurrent.Semaphore$NonfairSync)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> 	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
> 
> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000 nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> 	- locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> 	at org.apache.spark.ContextCleaner.org <http://org.apache.spark.contextcleaner.org/>$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
> 	at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
> 
> "DataStreamer for file /spark-logs/application_1427416187821_0004.inprogress block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait() [0x00007fd39ae44000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
> 	- locked <0x00000000e6994f30> (a java.util.LinkedList)
> 
> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0 tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e6995e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800 nid=0x3fb7 runnable [0x00007fd39b246000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e6886b58> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Yarn application state monitor" #73 daemon prio=5 os_prio=0 tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
> 
> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000 nid=0x3f77 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f waiting on condition [0x00007fd39b547000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e677ae98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "LeaseRenewer:hadoop@10.209.23.74:9000 <http://LeaseRenewer:hadoop@10.209.23.74:9000/>" #68 daemon prio=5 os_prio=0 tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
> 	at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> 	at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e runnable [0x00007fd3b80ff000]
>    java.lang.Thread.State: RUNNABLE
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "IPC Client (1489870911) connection to /10.209.23.74:9022 <http://10.209.23.74:9022/> from hadoop" #62 daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in Object.wait() [0x00007fd3b8501000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
> 	- locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
> 	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
> 
> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0 tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e66e9230> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 	at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> 	at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
> 
> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in Object.wait() [0x00007fd3b8703000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x00000000e66e8190> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> 
> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000 nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040 <http://SelectChannelConnector@0.0.0.0:4040/>" #54 daemon prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable [0x00007fd3b8fb6000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> 	- locked <0x00000000e671a620> (a java.lang.Object)
> 	at org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0 tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
> 	- locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> 	at org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0 tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x00000000e66609f0> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> 
> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000 nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800 nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800 nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800 nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800 nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800 nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000 nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920 <http://SocketConnector@0.0.0.0:53920/>" #42 daemon prio=5 os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable [0x00007fd3ba0de000]
>    java.lang.Thread.State: RUNNABLE
> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0 tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x00000000e6601270> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> 
> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0 tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x00000000e6600ca8> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> 
> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0 tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x00000000e65380d8> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> 
> "New I/O server boss #6" #36 daemon prio=5 os_prio=0 tid=0x00007fd39c04b800 nid=0x3df8 runnable [0x00007fd3ba6e2000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
> 	- locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800 nid=0x3df7 runnable [0x00007fd3ba7e3000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
> 	- locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800 nid=0x3df6 runnable [0x00007fd3ba8e4000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
> 	- locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000 nid=0x3df5 runnable [0x00007fd3ba9e5000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
> 	- locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800 nid=0x3df4 runnable [0x00007fd3baae6000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
> 	- locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000 nid=0x3df3 runnable [0x00007fd3babe7000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 	- locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
> 	- locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5 os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition [0x00007fd3baef9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5 os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition [0x00007fd3baffa000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0 tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0 tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0 tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0 tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800 nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000 nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800 nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800 nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000 nid=0x3d72 waiting on condition [0x00007fd40c460000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000 nid=0x3d71 waiting on condition [0x00007fd40c561000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000 nid=0x3d70 waiting on condition [0x00007fd40c662000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131 <http://SocketConnector@0.0.0.0:53131/>" #14 daemon prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable [0x00007fd40c966000]
>    java.lang.Thread.State: RUNNABLE
> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000 nid=0x3d67 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800 nid=0x3d66 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000 nid=0x3d65 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000 nid=0x3d64 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800 nid=0x3d63 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62 in Object.wait() [0x00007fd40f28e000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> 	- locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> 
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800 nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> 	- locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
> 
> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in Object.wait() [0x00007fd438220000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> 	- locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
> 	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
> 	at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
> 	at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
> 	at org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
> 	at org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
> 	at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
> 	at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
> 	at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
> 	at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
> 	at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
> 	at $line24.$read$$iwC$$iwC.<init>(<console>:35)
> 	at $line24.$read$$iwC.<init>(<console>:37)
> 	at $line24.$read.<init>(<console>:39)
> 	at $line24.$read$.<init>(<console>:43)
> 	at $line24.$read$.<clinit>(<console>)
> 	at $line24.$eval$.<init>(<console>:7)
> 	at $line24.$eval$.<clinit>(<console>)
> 	at $line24.$eval.$print(<console>)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:497)
> 	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> 	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
> 	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> 	at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
> 	at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
> 	at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
> 	at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
> 	at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> 	at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
> 	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
> 	at org.apache.spark.repl.Main$.main(Main.scala:31)
> 	at org.apache.spark.repl.Main.main(Main.scala)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:497)
> 	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
> 	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
> 	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
> 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
> 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> 
> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
> 
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000 nid=0x3d5c runnable
> 
> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800 nid=0x3d5d runnable
> 
> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800 nid=0x3d5e runnable
> 
> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000 nid=0x3d5f runnable
> 
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68 waiting on condition
> 
> JNI global references: 420
> 
> 
> 
> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <michael@databricks.com <ma...@databricks.com>> wrote:
> Is it possible to jstack the executors and see where they are hanging?
> 
> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jon.chase@gmail.com <ma...@gmail.com>> wrote:
> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB), executor memory 20GB, driver memory 10GB
> 
> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data spread out over roughly 2,000 Parquet files and my queries frequently hang. Simple queries like "select count(*) from ..." on the entire data set work ok.  Slightly more demanding ones with group by's and some aggregate functions (percentile_approx, avg, etc.) work ok as well, as long as I have some criteria in my where clause to keep the number of rows down.  
> 
> Once I hit some limit on query complexity and rows processed, my queries start to hang.  I've left them for up to an hour without seeing any progress.  No OOM's either - the job is just stuck.
> 
> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but with the same results: usually near the end of the tasks (like 780 of 800 complete), progress just stops:
> 
> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in stage 1.0 (TID 1618) in 800 ms on ip-10-209-22-211.eu-west-1.compute.internal (748/800)
> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu-west-1.compute.internal (749/800)
> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu-west-1.compute.internal (750/800)
> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu-west-1.compute.internal (751/800)
> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu-west-1.compute.internal (752/800)
> 
> ^^^^^^^ this is where it stays forever
> 
> Looking at the Spark UI, several of the executors still list active tasks.  I do see that the Shuffle Read for executors that don't have any tasks remaining is around 100MB, whereas it's more like 10MB for the executors that still have tasks.  
> 
> The first stage, mapPartitions, always completes fine.  It's the second stage (takeOrdered), that hangs.
> 
> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also encountered it when using JSON files (instead of Parquet).
> 
> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do are having this issue.
> 
> 


Re: Spark SQL queries hang forever

Posted by Jon Chase <jo...@gmail.com>.
Here are the jstacks from a 3 worker cluster.  Two of the workers are
sitting with 4 active tasks, the other has no active tasks.  Not sure if it
makes a difference, but I only see an ExecutorLauncher instance via JPS on
the worker that has no active tasks.

Here's the first stack dump, this machine has 4 active tasks:

2015-03-27 02:11:46
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):

"Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
nid=0x43a3 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
[0x00007f91bad9a000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000589884fc0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
nid=0x42e2 runnable [0x00007f91b4c54000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005898aa000> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
nid=0x42e1 runnable [0x00007f91b5d56000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058ad5b5f8> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
nid=0x42e0 runnable [0x00007f91b5e57000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005898aa0c0> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
nid=0x42df runnable [0x00007f91b5f58000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058ad5b6b8> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"Executor task launch worker-4" #71 daemon prio=5 os_prio=0
tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006cfda6800> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
at java.lang.Thread.run(Thread.java:745)

"Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
waiting on condition [0x00007f91b6c5b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000058af8e0b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
runnable [0x00007f91baf9c000]
   java.lang.Thread.State: RUNNABLE
at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
at
org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
at
org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0 tid=0x00007f91d8118800
nid=0x41ee waiting on condition [0x00007f91bae9b000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

"shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
nid=0x41ed runnable [0x00007f91baa97000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000589224af0> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"Executor task launch worker-2" #58 daemon prio=5 os_prio=0
tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006c8837750> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Executor task launch worker-1" #57 daemon prio=5 os_prio=0
tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006cc53bff0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Executor task launch worker-0" #56 daemon prio=5 os_prio=0
tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006cfda69b0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)

"shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
nid=0x41c6 runnable [0x00007f91bbac7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058984ee60> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005898f7290> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000058974cb00> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x000000058974cb00> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000589904038> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x0000000589904038> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000589884e80> (a
scala.concurrent.forkjoin.ForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800
nid=0x41be waiting on condition [0x00007f91ba370000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
at
org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)

"New I/O server boss #12" #39 daemon prio=5 os_prio=0
tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058992f188> (a sun.nio.ch.Util$2)
- locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
nid=0x41bc runnable [0x00007f91b9665000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000589931218> (a sun.nio.ch.Util$2)
- locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
nid=0x41bb runnable [0x00007f91b9766000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
- locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
nid=0x41ba runnable [0x00007f91b9867000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
- locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
nid=0x41b9 runnable [0x00007f91b9968000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
- locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
nid=0x41b8 runnable [0x00007f91b9c69000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
- locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5
os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
[0x00007f91b9d6a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000589884f20> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at
scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5
os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
[0x00007f91b9e6b000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000589884f20> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
at
akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
nid=0x419e runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
nid=0x419d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
nid=0x419c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
nid=0x419b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
nid=0x419a runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198 in
Object.wait() [0x00007f91d4254000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
nid=0x4197 in Object.wait() [0x00007f91d4355000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
condition [0x00007f91f1fdf000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000589a1b5f0> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at
akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
at
akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.ready(package.scala:86)
at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)

"VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
nid=0x4191 runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
nid=0x4192 runnable

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
nid=0x4193 runnable

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
nid=0x4194 runnable

"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
nid=0x4195 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
waiting on condition

JNI global references: 319












Here's the next worker, also for active tasks:

2015-03-27 02:12:54
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):

"Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
nid=0x44f1 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
[0x00007fcbbf87f000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000589510428> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"Executor task launch worker-8" #75 daemon prio=5 os_prio=0
tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006126cd358> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
nid=0x4459 runnable [0x00007fcbc88c7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058a6b8c98> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
nid=0x4458 runnable [0x00007fcbc7897000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058a6e5a18> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
nid=0x4457 runnable [0x00007fcbc7998000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058aade110> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
nid=0x4456 runnable [0x00007fcbc7b9a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058aade1d0> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"Executor task launch worker-7" #74 daemon prio=5 os_prio=0
tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000615539dd8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Executor task launch worker-5" #72 daemon prio=5 os_prio=0
tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000617bd4238> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Executor task launch worker-4" #71 daemon prio=5 os_prio=0
tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006126cd538> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
at
org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
at java.lang.Thread.run(Thread.java:745)

"Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
waiting on condition [0x00007fcbc4586000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000058adabe70> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
runnable [0x00007fcbc7e9d000]
   java.lang.Thread.State: RUNNABLE
at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
at
org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
at
org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007fcbf9608800
nid=0x4365 waiting on condition [0x00007fcbc89c8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

"shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
nid=0x4364 runnable [0x00007fcbc8ac9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000589116ae0> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
nid=0x4341 waiting on condition [0x00007fcbc95f4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)

"shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
nid=0x4340 runnable [0x00007fcbc94f3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058964c068> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000058a6fa058> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000058928c438> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x000000058928c438> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x000000058928c5a0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000
nid=0x433a waiting on condition [0x00007fcbc85c4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
at
org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)

"ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000058a70a468> (a
scala.concurrent.forkjoin.ForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"New I/O server boss #12" #41 daemon prio=5 os_prio=0
tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
- locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
nid=0x4336 runnable [0x00007fcbc6f90000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
- locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
nid=0x4335 runnable [0x00007fcbc7091000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
- locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
nid=0x4334 runnable [0x00007fcbc7392000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058a796550> (a sun.nio.ch.Util$2)
- locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
nid=0x4333 runnable [0x00007fcbc7493000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
- locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
nid=0x4332 runnable [0x00007fcbc7594000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
- locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
[0x00007fcbc7695000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005893c1820> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at
scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
[0x00007fcbc7796000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005893c1820> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
at
akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
nid=0x4318 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
nid=0x4317 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
nid=0x4316 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
nid=0x4315 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
nid=0x4314 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312 in
Object.wait() [0x00007fcbe1be7000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
nid=0x4311 in Object.wait() [0x00007fcbe8150000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
condition [0x00007fcbff9b9000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000589585398> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at
akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
at
akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.ready(package.scala:86)
at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)

"VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
nid=0x430b runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
nid=0x430c runnable

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
nid=0x430d runnable

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
nid=0x430e runnable

"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
nid=0x430f runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
waiting on condition

JNI global references: 375










And the final worker, no active tasks:


2015-03-27 02:13:26
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):

"Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
nid=0x3c5b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
[0x00007f36d2e33000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005895676e8> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
nid=0x3b88 runnable [0x00007f36dafb9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058a86a7d8> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
nid=0x3b87 runnable [0x00007f36d7138000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058abea6c8> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
nid=0x3b86 runnable [0x00007f36daeb8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058a86a898> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
nid=0x3b85 runnable [0x00007f36d7e3b000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058abea788> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
at java.lang.Thread.run(Thread.java:745)

"Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
waiting on condition [0x00007f36d7b3a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000058aecd008> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
runnable [0x00007f36db4be000]
   java.lang.Thread.State: RUNNABLE
at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
at
org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
at
org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007f36f8025000
nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

"shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
nid=0x3a96 runnable [0x00007f36dc0ea000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000589202c58> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
nid=0x3a6d waiting on condition [0x00007f36dcc15000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)

"shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
nid=0x3a6c runnable [0x00007f36dcb14000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058a825c18> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005897334d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000005896b0d20> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000005896b0dc8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000
nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
at
org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)

"ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005895675a8> (a
scala.concurrent.forkjoin.ForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"New I/O server boss #12" #41 daemon prio=5 os_prio=0
tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058970f538> (a sun.nio.ch.Util$2)
- locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
nid=0x3a62 runnable [0x00007f36da5b1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
- locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
nid=0x3a61 runnable [0x00007f36da6b2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
- locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
nid=0x3a60 runnable [0x00007f36da7b3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
- locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
nid=0x3a5f runnable [0x00007f36da8b4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
- locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
nid=0x3a5e runnable [0x00007f36da9b5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
- locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
[0x00007f36dacb6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000589567648> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at
scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
[0x00007f36dadb7000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000589567648> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
at
akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
nid=0x3a43 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
nid=0x3a42 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
nid=0x3a41 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
nid=0x3a40 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
nid=0x3a3f runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d in
Object.wait() [0x00007f36f51e5000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
condition [0x00007f3712fda000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000058a974ab8> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at
akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
at
akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.ready(package.scala:86)
at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)

"VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
nid=0x3a34 runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
nid=0x3a35 runnable

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
nid=0x3a36 runnable

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
nid=0x3a37 runnable

"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
nid=0x3a3a runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
waiting on condition

JNI global references: 486








And the driver:

2015-03-27 02:20:47
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):

"qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e67194f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
nid=0x465a waiting on condition [0x00007fd3b8eb5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e67194f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e67194f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
nid=0x41d0 waiting on condition [0x00007fd38feef000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e67194f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000fd545eb8> (a
scala.concurrent.forkjoin.ForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5 os_prio=0
tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition [0x00007fd3b8400000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e64f49f8> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
nid=0x417f waiting on condition [0x00007fd398935000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e67194f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800
nid=0x409c waiting on condition [0x00007fd3902f1000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e66e9a00> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000
nid=0x409b waiting on condition [0x00007fd3903f2000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e66e9a00> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800
nid=0x409a waiting on condition [0x00007fd3904f3000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e66e9a00> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800
nid=0x4098 waiting on condition [0x00007fd3905f4000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e66e9a00> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
nid=0x4092 runnable [0x00007fd398b37000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e685f320> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
nid=0x4091 runnable [0x00007fd398a36000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e6871c30> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
nid=0x4090 runnable [0x00007fd39903a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e6873520> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
at java.lang.Thread.run(Thread.java:745)

"ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e7f19070> (a
scala.concurrent.forkjoin.ForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0
tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e64f49f8> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e66fe220> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e740fa58> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e740fce8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
at java.lang.Thread.run(Thread.java:745)

"BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e737a0c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e737add8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
at java.lang.Thread.run(Thread.java:745)

"Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
at
com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)

"ResponseProcessor for block
BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
[0x00007fd39a941000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
- locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)

"SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
nid=0x3fda waiting on condition [0x00007fd39aa42000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e58145f0> (a
java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)

"Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000
nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
at
org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
at
org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
at
org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
at org.apache.spark.ContextCleaner.org
$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)

"DataStreamer for file
/spark-logs/application_1427416187821_0004.inprogress block
BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon
prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait()
[0x00007fd39ae44000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
- locked <0x00000000e6994f30> (a java.util.LinkedList)

"metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e6995e88> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
nid=0x3fb7 runnable [0x00007fd39b246000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e6886b58> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)

"Yarn application state monitor" #73 daemon prio=5 os_prio=0
tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)

"Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
nid=0x3f77 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f
waiting on condition [0x00007fd39b547000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e677ae98> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
at java.lang.Thread.run(Thread.java:745)

"Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e
runnable [0x00007fd3b80ff000]
   java.lang.Thread.State: RUNNABLE
at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
at
org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
at
org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
at java.lang.Thread.run(Thread.java:745)

"IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop" #62
daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in Object.wait()
[0x00007fd3b8501000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
- locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)

"dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e66e9230> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
at
java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)

"Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in
Object.wait() [0x00007fd3b8703000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000e66e8190> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e67194f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54 daemon
prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
[0x00007fd3b8fb6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000000e671a620> (a java.lang.Object)
at
org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

"qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
- locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

"SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000e66609f0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e6539dc8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e6539dc8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e6539dc8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e6539dc8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e6539dc8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e6539dc8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e6539dc8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon prio=5
os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable [0x00007fd3ba0de000]
   java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at
org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
at
org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

"BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000e6601270> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000e6600ca8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000e65380d8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"New I/O server boss #6" #36 daemon prio=5 os_prio=0 tid=0x00007fd39c04b800
nid=0x3df8 runnable [0x00007fd3ba6e2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
- locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
nid=0x3df7 runnable [0x00007fd3ba7e3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
- locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
nid=0x3df6 runnable [0x00007fd3ba8e4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
- locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
nid=0x3df5 runnable [0x00007fd3ba9e5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
- locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
nid=0x3df4 runnable [0x00007fd3baae6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
- locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
nid=0x3df3 runnable [0x00007fd3babe7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
- locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
[0x00007fd3baef9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e64bbb40> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at
scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
[0x00007fd3baffa000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e64bbb40> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0
tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e64f49f8> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at
scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0
tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e64f49f8> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0
tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e64f49f8> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
at
akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)

"qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e42808f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
nid=0x3d75 waiting on condition [0x00007fd40c15d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e42808f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
nid=0x3d74 waiting on condition [0x00007fd40c25e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e42808f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
nid=0x3d73 waiting on condition [0x00007fd40c35f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e42808f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
nid=0x3d72 waiting on condition [0x00007fd40c460000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e42808f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
nid=0x3d71 waiting on condition [0x00007fd40c561000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e42808f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
nid=0x3d70 waiting on condition [0x00007fd40c662000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000e42808f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

"qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon
prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable
[0x00007fd40c966000]
   java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at
org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
at
org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
nid=0x3d67 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
nid=0x3d66 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
nid=0x3d65 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
nid=0x3d64 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
nid=0x3d63 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62 in
Object.wait() [0x00007fd40f28e000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
Object.wait() [0x00007fd438220000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
- locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
at
org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
at
org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
at
$line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
at $line24.$read$$iwC$$iwC.<init>(<console>:35)
at $line24.$read$$iwC.<init>(<console>:37)
at $line24.$read.<init>(<console>:39)
at $line24.$read$.<init>(<console>:43)
at $line24.$read$.<clinit>(<console>)
at $line24.$eval$.<init>(<console>:7)
at $line24.$eval$.<clinit>(<console>)
at $line24.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
at
org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
at org.apache.spark.repl.SparkILoop.org
$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
at
scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org
$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

"VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000 nid=0x3d5c
runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800 nid=0x3d5d
runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800 nid=0x3d5e
runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000 nid=0x3d5f
runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
waiting on condition

JNI global references: 420



On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <mi...@databricks.com>
wrote:

> Is it possible to jstack the executors and see where they are hanging?
>
> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:
>
>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB),
>> executor memory 20GB, driver memory 10GB
>>
>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data spread
>> out over roughly 2,000 Parquet files and my queries frequently hang. Simple
>> queries like "select count(*) from ..." on the entire data set work ok.
>> Slightly more demanding ones with group by's and some aggregate functions
>> (percentile_approx, avg, etc.) work ok as well, as long as I have some
>> criteria in my where clause to keep the number of rows down.
>>
>> Once I hit some limit on query complexity and rows processed, my queries
>> start to hang.  I've left them for up to an hour without seeing any
>> progress.  No OOM's either - the job is just stuck.
>>
>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but
>> with the same results: usually near the end of the tasks (like 780 of 800
>> complete), progress just stops:
>>
>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in
>> stage 1.0 (TID 1618) in 800 ms on
>> ip-10-209-22-211.eu-west-1.compute.internal (748/800)
>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in
>> stage 1.0 (TID 1623) in 622 ms on
>> ip-10-105-12-41.eu-west-1.compute.internal (749/800)
>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in
>> stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu-west-1.compute.internal
>> (750/800)
>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in
>> stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu-west-1.compute.internal
>> (751/800)
>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in
>> stage 1.0 (TID 1625) in 669 ms on
>> ip-10-105-12-41.eu-west-1.compute.internal (752/800)
>>
>> ^^^^^^^ this is where it stays forever
>>
>> Looking at the Spark UI, several of the executors still list active
>> tasks.  I do see that the Shuffle Read for executors that don't have any
>> tasks remaining is around 100MB, whereas it's more like 10MB for the
>> executors that still have tasks.
>>
>> The first stage, mapPartitions, always completes fine.  It's the second
>> stage (takeOrdered), that hangs.
>>
>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
>> encountered it when using JSON files (instead of Parquet).
>>
>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
>> are having this issue.
>>
>
>

Re: Spark SQL queries hang forever

Posted by Michael Armbrust <mi...@databricks.com>.
Is it possible to jstack the executors and see where they are hanging?

On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:

> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB),
> executor memory 20GB, driver memory 10GB
>
> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data spread
> out over roughly 2,000 Parquet files and my queries frequently hang. Simple
> queries like "select count(*) from ..." on the entire data set work ok.
> Slightly more demanding ones with group by's and some aggregate functions
> (percentile_approx, avg, etc.) work ok as well, as long as I have some
> criteria in my where clause to keep the number of rows down.
>
> Once I hit some limit on query complexity and rows processed, my queries
> start to hang.  I've left them for up to an hour without seeing any
> progress.  No OOM's either - the job is just stuck.
>
> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but
> with the same results: usually near the end of the tasks (like 780 of 800
> complete), progress just stops:
>
> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in
> stage 1.0 (TID 1618) in 800 ms on
> ip-10-209-22-211.eu-west-1.compute.internal (748/800)
> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in
> stage 1.0 (TID 1623) in 622 ms on
> ip-10-105-12-41.eu-west-1.compute.internal (749/800)
> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in
> stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu-west-1.compute.internal
> (750/800)
> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in
> stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu-west-1.compute.internal
> (751/800)
> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in
> stage 1.0 (TID 1625) in 669 ms on
> ip-10-105-12-41.eu-west-1.compute.internal (752/800)
>
> ^^^^^^^ this is where it stays forever
>
> Looking at the Spark UI, several of the executors still list active
> tasks.  I do see that the Shuffle Read for executors that don't have any
> tasks remaining is around 100MB, whereas it's more like 10MB for the
> executors that still have tasks.
>
> The first stage, mapPartitions, always completes fine.  It's the second
> stage (takeOrdered), that hangs.
>
> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
> encountered it when using JSON files (instead of Parquet).
>
> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do are
> having this issue.
>