Re: Spark SQL queries hang forever

[Reynold Xin <rx...@databricks.com>]

Thanks for the information.

Adding the dev list.

Do you mind trying 1.3.1 to see if the issue is fixed?  I could be wrong
but it migtht be related to https://issues.apache.org/jira/browse/SPARK-6578



On Thu, Apr 16, 2015 at 2:34 AM, Michael Allman <mi...@videoamp.com>
wrote:

> After many, many manual (and automated) trials using a spark sql query
> that *sometimes* triggers the hang I found the following:
>
>
>    - When I set spark.shuffle.blockTransferService to netty (the
>    default), I can sometimes reproduce the problem.
>    - When I set spark.shuffle.blockTransferService to nio (the Spark 1.1
>    default), I have not yet been able to reproduce the problem.
>
>
> While far from a reliable conclusion, the workaround I'm going with for
> now is setting "spark.shuffle.blockTransferService nio" in
> spark-defaults.conf. One of the reasons I homed in on this setting as a
> possible trigger was the fact that its default value changed starting in
> Spark 1.2. We haven't experienced a similar problem in our current Spark
> 1.1 cluster, and we're skipping directly to Spark 1.3. Has anyone else
> tried this apparent workaround?
>
> Cheers,
>
> Michael
>
> On Apr 15, 2015, at 11:23 PM, Michael Allman <mi...@videoamp.com> wrote:
>
> FWIW I'm attempting an upgrade from Spark 1.1 to Spark 1.3 and I'm seeing
> this exact problem in ShuffleBlockFetcherIterator in the query takeOrdered
> stage. Sometimes a single task hangs, sometimes a couple dozen. The final
> stage has 200 tasks. I'm using the Spark standalone scheduler, parquet
> files. Here's the stack trace of the stuck thread:
>
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:389)
>
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:154)
>
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:149)
> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> org.apache.spark.scheduler.Task.run(Task.scala:64)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
>
> Anyone find a reliable fix/workaround? I'm going to try some things
> myself. If I find something that works I'll post back.
>
> Cheers,
>
> Michael
>
>
> On Mar 26, 2015, at 7:21 PM, Jon Chase <jo...@gmail.com> wrote:
>
> Here are the jstacks from a 3 worker cluster.  Two of the workers are
> sitting with 4 active tasks, the other has no active tasks.  Not sure if it
> makes a difference, but I only see an ExecutorLauncher instance via JPS on
> the worker that has no active tasks.
>
> Here's the first stack dump, this machine has 4 active tasks:
>
> 2015-03-27 02:11:46
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
> nid=0x43a3 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
> os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
> [0x00007f91bad9a000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884fc0> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
> nid=0x42e2 runnable [0x00007f91b4c54000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005898aa000> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
> nid=0x42e1 runnable [0x00007f91b5d56000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058ad5b5f8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
> nid=0x42e0 runnable [0x00007f91b5e57000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005898aa0c0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
> nid=0x42df runnable [0x00007f91b5f58000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058ad5b6b8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cfda6800> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
> tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
> waiting on condition [0x00007f91b6c5b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058af8e0b8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
> runnable [0x00007f91baf9c000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0
> tid=0x00007f91d8118800 nid=0x41ee waiting on condition [0x00007f91bae9b000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
> nid=0x41ed runnable [0x00007f91baa97000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589224af0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
> tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006c8837750> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
> tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cc53bff0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
> tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006cfda69b0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
> nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
> nid=0x41c6 runnable [0x00007f91bbac7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058984ee60> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
> tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005898f7290> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
> tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058974cb00> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
> tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000589904038> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x0000000589904038> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
> tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884e80> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800
> nid=0x41be waiting on condition [0x00007f91ba370000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O server boss #12" #39 daemon prio=5 os_prio=0
> tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
> - locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
> nid=0x41bc runnable [0x00007f91b9665000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
> - locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
> nid=0x41bb runnable [0x00007f91b9766000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
> - locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
> nid=0x41ba runnable [0x00007f91b9867000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
> nid=0x41b9 runnable [0x00007f91b9968000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
> - locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
> nid=0x41b8 runnable [0x00007f91b9c69000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
> - locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5
> os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
> [0x00007f91b9d6a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884f20> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5
> os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
> [0x00007f91b9e6b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589884f20> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
> nid=0x419e runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
> nid=0x419d waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
> nid=0x419c waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
> nid=0x419b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
> nid=0x419a runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198
> in Object.wait() [0x00007f91d4254000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
> nid=0x4197 in Object.wait() [0x00007f91d4355000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
> condition [0x00007f91f1fdf000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589a1b5f0> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
> nid=0x4191 runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
> nid=0x4192 runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
> nid=0x4193 runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
> nid=0x4194 runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
> nid=0x4195 runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
> waiting on condition
>
> JNI global references: 319
>
>
>
>
>
>
>
>
>
>
>
>
> Here's the next worker, also for active tasks:
>
> 2015-03-27 02:12:54
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
> nid=0x44f1 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
> os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
> [0x00007fcbbf87f000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589510428> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
> tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006126cd358> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
> nid=0x4459 runnable [0x00007fcbc88c7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a6b8c98> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
> nid=0x4458 runnable [0x00007fcbc7897000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a6e5a18> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
> nid=0x4457 runnable [0x00007fcbc7998000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058aade110> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
> nid=0x4456 runnable [0x00007fcbc7b9a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058aade1d0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
> tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000615539dd8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
> tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000617bd4238> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
> tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006126cd538> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
> tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
> waiting on condition [0x00007fcbc4586000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058adabe70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
> runnable [0x00007fcbc7e9d000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> tid=0x00007fcbf9608800 nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
> nid=0x4364 runnable [0x00007fcbc8ac9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589116ae0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
> nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
> nid=0x4340 runnable [0x00007fcbc94f3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058964c068> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a6fa058> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058928c438> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058928c438> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x000000058928c5a0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000
> nid=0x433a waiting on condition [0x00007fcbc85c4000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a70a468> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
> - locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
> nid=0x4336 runnable [0x00007fcbc6f90000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
> - locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
> nid=0x4335 runnable [0x00007fcbc7091000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
> - locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
> nid=0x4334 runnable [0x00007fcbc7392000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
> - locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
> nid=0x4333 runnable [0x00007fcbc7493000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
> - locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
> nid=0x4332 runnable [0x00007fcbc7594000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
> - locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
> [0x00007fcbc7695000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005893c1820> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
> [0x00007fcbc7796000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005893c1820> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
> nid=0x4318 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
> nid=0x4317 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
> nid=0x4316 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
> nid=0x4315 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
> nid=0x4314 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312
> in Object.wait() [0x00007fcbe1be7000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
> nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
> condition [0x00007fcbff9b9000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589585398> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
> nid=0x430b runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
> nid=0x430c runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
> nid=0x430d runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
> nid=0x430e runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
> nid=0x430f runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
> waiting on condition
>
> JNI global references: 375
>
>
>
>
>
>
>
>
>
>
> And the final worker, no active tasks:
>
>
> 2015-03-27 02:13:26
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
> nid=0x3c5b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
> os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
> [0x00007f36d2e33000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005895676e8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
> nid=0x3b88 runnable [0x00007f36dafb9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a86a7d8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
> nid=0x3b87 runnable [0x00007f36d7138000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058abea6c8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
> nid=0x3b86 runnable [0x00007f36daeb8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a86a898> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
> nid=0x3b85 runnable [0x00007f36d7e3b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058abea788> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
> tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
> waiting on condition [0x00007f36d7b3a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058aecd008> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
> runnable [0x00007f36db4be000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
> tid=0x00007f36f8025000 nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
> nid=0x3a96 runnable [0x00007f36dc0ea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000589202c58> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
> nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>
> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
> nid=0x3a6c runnable [0x00007f36dcb14000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058a825c18> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
> tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005897334d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
> tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000005896b0d20> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
> tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000
> nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
> tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005895675a8> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
> tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
> - locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
> nid=0x3a62 runnable [0x00007f36da5b1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
> - locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
> nid=0x3a61 runnable [0x00007f36da6b2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
> - locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
> nid=0x3a60 runnable [0x00007f36da7b3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
> - locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
> nid=0x3a5f runnable [0x00007f36da8b4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
> - locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
> nid=0x3a5e runnable [0x00007f36da9b5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
> - locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5
> os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
> [0x00007f36dacb6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589567648> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5
> os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
> [0x00007f36dadb7000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000589567648> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
> tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
> nid=0x3a43 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
> nid=0x3a42 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
> nid=0x3a41 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
> nid=0x3a40 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
> nid=0x3a3f runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d
> in Object.wait() [0x00007f36f51e5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
> nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
> condition [0x00007f3712fda000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000058a974ab8> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>
> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
> nid=0x3a34 runnable
>
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
> nid=0x3a35 runnable
>
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
> nid=0x3a36 runnable
>
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
> nid=0x3a37 runnable
>
> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
> nid=0x3a3a runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
> waiting on condition
>
> JNI global references: 486
>
>
>
>
>
>
>
>
> And the driver:
>
> 2015-03-27 02:20:47
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>
> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
> nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
> nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
> nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
> nid=0x41d0 waiting on condition [0x00007fd38feef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
> tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000fd545eb8> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5
> os_prio=0 tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition
> [0x00007fd3b8400000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
> nid=0x417f waiting on condition [0x00007fd398935000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800
> nid=0x409c waiting on condition [0x00007fd3902f1000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000
> nid=0x409b waiting on condition [0x00007fd3903f2000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800
> nid=0x409a waiting on condition [0x00007fd3904f3000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800
> nid=0x4098 waiting on condition [0x00007fd3905f4000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9a00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
> tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
> nid=0x4092 runnable [0x00007fd398b37000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e685f320> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
> nid=0x4091 runnable [0x00007fd398a36000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6871c30> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
> nid=0x4090 runnable [0x00007fd39903a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6873520> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
> tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
> at java.lang.Thread.run(Thread.java:745)
>
> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
> tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e7f19070> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0
> tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
> tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66fe220> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
> tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e740fa58> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
> tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e740fce8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
> tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
> tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e737a0c8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
> tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e737add8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
> tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
> at java.lang.Thread.run(Thread.java:745)
>
> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
> tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
> at
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>
> "ResponseProcessor for block
> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
> prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
> [0x00007fd39a941000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
> - locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>
> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
> nid=0x3fda waiting on condition [0x00007fd39aa42000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e58145f0> (a
> java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>
> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000
> nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> at
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at org.apache.spark.ContextCleaner.org
> <http://org.apache.spark.contextcleaner.org/>
> $apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
> at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>
> "DataStreamer for file
> /spark-logs/application_1427416187821_0004.inprogress block
> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon
> prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait()
> [0x00007fd39ae44000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
> - locked <0x00000000e6994f30> (a java.util.LinkedList)
>
> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
> tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6995e88> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
> nid=0x3fb7 runnable [0x00007fd39b246000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6886b58> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
>
> "Yarn application state monitor" #73 daemon prio=5 os_prio=0
> tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>
> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
> nid=0x3f77 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f
> waiting on condition [0x00007fd39b547000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e677ae98> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
> tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e
> runnable [0x00007fd3b80ff000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
> at java.lang.Thread.run(Thread.java:745)
>
> "IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop"
> #62 daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in
> Object.wait() [0x00007fd3b8501000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
> - locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>
> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
> tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e66e9230> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> at
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>
> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in
> Object.wait() [0x00007fd3b8703000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e66e8190> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
> nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e67194f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54
> daemon prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
> [0x00007fd3b8fb6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> - locked <0x00000000e671a620> (a java.lang.Object)
> at
> org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
> tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> at
> org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
> tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e66609f0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
> nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
> nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
> nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
> nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
> nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
> nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
> nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e6539dc8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon
> prio=5 os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable
> [0x00007fd3ba0de000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
> tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e6601270> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
> tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
> tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000e65380d8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
>
> "New I/O server boss #6" #36 daemon prio=5 os_prio=0
> tid=0x00007fd39c04b800 nid=0x3df8 runnable [0x00007fd3ba6e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
> nid=0x3df7 runnable [0x00007fd3ba7e3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
> nid=0x3df6 runnable [0x00007fd3ba8e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
> - locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
> nid=0x3df5 runnable [0x00007fd3ba9e5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
> - locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
> nid=0x3df4 runnable [0x00007fd3baae6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
> - locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
> nid=0x3df3 runnable [0x00007fd3babe7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
> - locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
> os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
> [0x00007fd3baef9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64bbb40> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
> os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
> [0x00007fd3baffa000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64bbb40> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0
> tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0
> tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0
> tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e64f49f8> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
> tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
> nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
> nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
> nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
> nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
> nid=0x3d72 waiting on condition [0x00007fd40c460000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
> nid=0x3d71 waiting on condition [0x00007fd40c561000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
> nid=0x3d70 waiting on condition [0x00007fd40c662000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000e42808f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon
> prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable
> [0x00007fd40c966000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
> at
> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
> nid=0x3d67 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
> nid=0x3d66 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
> nid=0x3d65 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
> nid=0x3d64 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
> nid=0x3d63 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62
> in Object.wait() [0x00007fd40f28e000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
> nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
> - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
> Object.wait() [0x00007fd438220000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> - locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
> at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
> at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
> at
> org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
> at
> org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
> at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
> at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
> at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
> at
> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
> at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
> at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
> at $line24.$read$$iwC$$iwC.<init>(<console>:35)
> at $line24.$read$$iwC.<init>(<console>:37)
> at $line24.$read.<init>(<console>:39)
> at $line24.$read$.<init>(<console>:43)
> at $line24.$read$.<clinit>(<console>)
> at $line24.$eval$.<init>(<console>:7)
> at $line24.$eval$.<clinit>(<console>)
> at $line24.$eval.$print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
> at org.apache.spark.repl.SparkILoop.org
> <http://org.apache.spark.repl.sparkiloop.org/>
> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.org
> <http://org.apache.spark.repl.sparkiloop.org/>
> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000
> nid=0x3d5c runnable
>
> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800
> nid=0x3d5d runnable
>
> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800
> nid=0x3d5e runnable
>
> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000
> nid=0x3d5f runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
> waiting on condition
>
> JNI global references: 420
>
>
>
> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> Is it possible to jstack the executors and see where they are hanging?
>>
>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jo...@gmail.com> wrote:
>>
>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB),
>>> executor memory 20GB, driver memory 10GB
>>>
>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data
>>> spread out over roughly 2,000 Parquet files and my queries frequently hang.
>>> Simple queries like "select count(*) from ..." on the entire data set work
>>> ok.  Slightly more demanding ones with group by's and some aggregate
>>> functions (percentile_approx, avg, etc.) work ok as well, as long as I have
>>> some criteria in my where clause to keep the number of rows down.
>>>
>>> Once I hit some limit on query complexity and rows processed, my queries
>>> start to hang.  I've left them for up to an hour without seeing any
>>> progress.  No OOM's either - the job is just stuck.
>>>
>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but
>>> with the same results: usually near the end of the tasks (like 780 of 800
>>> complete), progress just stops:
>>>
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in
>>> stage 1.0 (TID 1618) in 800 ms on ip-10-209-22-211.eu-west-1.compute.internal
>>> (748/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in
>>> stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>> (749/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in
>>> stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>> (750/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in
>>> stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu-west-1.compute.internal
>>> (751/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in
>>> stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu-west-1.compute.internal
>>> (752/800)
>>>
>>> ^^^^^^^ this is where it stays forever
>>>
>>> Looking at the Spark UI, several of the executors still list active
>>> tasks.  I do see that the Shuffle Read for executors that don't have any
>>> tasks remaining is around 100MB, whereas it's more like 10MB for the
>>> executors that still have tasks.
>>>
>>> The first stage, mapPartitions, always completes fine.  It's the second
>>> stage (takeOrdered), that hangs.
>>>
>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
>>> encountered it when using JSON files (instead of Parquet).
>>>
>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
>>> are having this issue.
>>>
>>
>>
>
>
>

Re: Spark SQL queries hang forever

[Michael Allman <mi...@videoamp.com>]

Hi Reynold,

Our build is based off of the 1.3 branch as of commit 6d3c4d8b04b2738a821dfcc3df55a5635b89e506. Unless your fix is on another branch it appears this problem still exists.

LMK if there's anything else I can help with.

Thanks,

Michael

> On Apr 16, 2015, at 11:44 AM, Reynold Xin <rx...@databricks.com> wrote:
> 
> Thanks for the information.
> 
> Adding the dev list.
> 
> Do you mind trying 1.3.1 to see if the issue is fixed?  I could be wrong but it migtht be related to https://issues.apache.org/jira/browse/SPARK-6578 <https://issues.apache.org/jira/browse/SPARK-6578>
> 
>  
> 
> On Thu, Apr 16, 2015 at 2:34 AM, Michael Allman <michael@videoamp.com <ma...@videoamp.com>> wrote:
> After many, many manual (and automated) trials using a spark sql query that *sometimes* triggers the hang I found the following:
> 
> When I set spark.shuffle.blockTransferService to netty (the default), I can sometimes reproduce the problem.
> When I set spark.shuffle.blockTransferService to nio (the Spark 1.1 default), I have not yet been able to reproduce the problem.
> 
> While far from a reliable conclusion, the workaround I'm going with for now is setting "spark.shuffle.blockTransferService nio" in spark-defaults.conf. One of the reasons I homed in on this setting as a possible trigger was the fact that its default value changed starting in Spark 1.2. We haven't experienced a similar problem in our current Spark 1.1 cluster, and we're skipping directly to Spark 1.3. Has anyone else tried this apparent workaround?
> 
> Cheers,
> 
> Michael
> 
>> On Apr 15, 2015, at 11:23 PM, Michael Allman <michael@videoamp.com <ma...@videoamp.com>> wrote:
>> 
>> FWIW I'm attempting an upgrade from Spark 1.1 to Spark 1.3 and I'm seeing this exact problem in ShuffleBlockFetcherIterator in the query takeOrdered stage. Sometimes a single task hangs, sometimes a couple dozen. The final stage has 200 tasks. I'm using the Spark standalone scheduler, parquet files. Here's the stack trace of the stuck thread:
>> 
>> sun.misc.Unsafe.park(Native Method)
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:389)
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:154)
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:149)
>> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> org.apache.spark.scheduler.Task.run(Task.scala:64)
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> java.lang.Thread.run(Thread.java:745)
>> 
>> Anyone find a reliable fix/workaround? I'm going to try some things myself. If I find something that works I'll post back.
>> 
>> Cheers,
>> 
>> Michael
>> 
>> 
>>> On Mar 26, 2015, at 7:21 PM, Jon Chase <jon.chase@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> Here are the jstacks from a 3 worker cluster.  Two of the workers are sitting with 4 active tasks, the other has no active tasks.  Not sure if it makes a difference, but I only see an ExecutorLauncher instance via JPS on the worker that has no active tasks.  
>>> 
>>> Here's the first stack dump, this machine has 4 active tasks:
>>> 
>>> 2015-03-27 02:11:46
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000 nid=0x43a3 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5 os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition [0x00007f91bad9a000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884fc0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000 nid=0x42e2 runnable [0x00007f91b4c54000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005898aa000> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800 nid=0x42e1 runnable [0x00007f91b5d56000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058ad5b5f8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000 nid=0x42e0 runnable [0x00007f91b5e57000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005898aa0c0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800 nid=0x42df runnable [0x00007f91b5f58000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058ad5b6b8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cfda6800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0 tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2 waiting on condition [0x00007f91b6c5b000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058af8e0b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef runnable [0x00007f91baf9c000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0 tid=0x00007f91d8118800 nid=0x41ee waiting on condition [0x00007f91bae9b000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000 nid=0x41ed runnable [0x00007f91baa97000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589224af0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0 tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006c8837750> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0 tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cc53bff0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0 tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cfda69b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800 nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800 nid=0x41c6 runnable [0x00007f91bbac7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058984ee60> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0 tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005898f7290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0 tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058974cb00> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058974cb00> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0 tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x0000000589904038> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x0000000589904038> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0 tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884e80> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800 nid=0x41be waiting on condition [0x00007f91ba370000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O server boss #12" #39 daemon prio=5 os_prio=0 tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058992f188> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000 nid=0x41bc runnable [0x00007f91b9665000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589931218> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800 nid=0x41bb runnable [0x00007f91b9766000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800 nid=0x41ba runnable [0x00007f91b9867000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800 nid=0x41b9 runnable [0x00007f91b9968000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000 nid=0x41b8 runnable [0x00007f91b9c69000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5 os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition [0x00007f91b9d6a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5 os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition [0x00007f91b9e6b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800 nid=0x419e runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000 nid=0x419d waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000 nid=0x419c waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000 nid=0x419b waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800 nid=0x419a runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198 in Object.wait() [0x00007f91d4254000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800 nid=0x4197 in Object.wait() [0x00007f91d4355000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on condition [0x00007f91f1fdf000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589a1b5f0> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000 nid=0x4191 runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800 nid=0x4192 runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800 nid=0x4193 runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000 nid=0x4194 runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000 nid=0x4195 runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f waiting on condition
>>> 
>>> JNI global references: 319
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Here's the next worker, also for active tasks:
>>> 
>>> 2015-03-27 02:12:54
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000 nid=0x44f1 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5 os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition [0x00007fcbbf87f000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589510428> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0 tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006126cd358> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800 nid=0x4459 runnable [0x00007fcbc88c7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a6b8c98> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000 nid=0x4458 runnable [0x00007fcbc7897000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a6e5a18> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800 nid=0x4457 runnable [0x00007fcbc7998000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058aade110> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000 nid=0x4456 runnable [0x00007fcbc7b9a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058aade1d0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0 tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000615539dd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0 tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000617bd4238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006126cd538> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0 tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369 waiting on condition [0x00007fcbc4586000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058adabe70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366 runnable [0x00007fcbc7e9d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007fcbf9608800 nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800 nid=0x4364 runnable [0x00007fcbc8ac9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589116ae0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000 nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000 nid=0x4340 runnable [0x00007fcbc94f3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058964c068> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a6fa058> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058928c438> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058928c438> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058928c5a0> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000 nid=0x433a waiting on condition [0x00007fcbc85c4000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a70a468> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000 nid=0x4336 runnable [0x00007fcbc6f90000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800 nid=0x4335 runnable [0x00007fcbc7091000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000 nid=0x4334 runnable [0x00007fcbc7392000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a796550> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000 nid=0x4333 runnable [0x00007fcbc7493000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000 nid=0x4332 runnable [0x00007fcbc7594000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition [0x00007fcbc7695000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition [0x00007fcbc7796000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800 nid=0x4318 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800 nid=0x4317 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000 nid=0x4316 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000 nid=0x4315 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000 nid=0x4314 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312 in Object.wait() [0x00007fcbe1be7000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800 nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on condition [0x00007fcbff9b9000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589585398> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000 nid=0x430b runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800 nid=0x430c runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800 nid=0x430d runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000 nid=0x430e runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000 nid=0x430f runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319 waiting on condition
>>> 
>>> JNI global references: 375
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> And the final worker, no active tasks:
>>> 
>>> 
>>> 2015-03-27 02:13:26
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000 nid=0x3c5b waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5 os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition [0x00007f36d2e33000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005895676e8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000 nid=0x3b88 runnable [0x00007f36dafb9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a86a7d8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800 nid=0x3b87 runnable [0x00007f36d7138000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058abea6c8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800 nid=0x3b86 runnable [0x00007f36daeb8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a86a898> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000 nid=0x3b85 runnable [0x00007f36d7e3b000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058abea788> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0 tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d waiting on condition [0x00007f36d7b3a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058aecd008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a runnable [0x00007f36db4be000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007f36f8025000 nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000 nid=0x3a96 runnable [0x00007f36dc0ea000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589202c58> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000 nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000 nid=0x3a6c runnable [0x00007f36dcb14000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a825c18> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005897334d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000005896b0d20> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000005896b0dc8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000 nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005895675a8> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058970f538> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800 nid=0x3a62 runnable [0x00007f36da5b1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800 nid=0x3a61 runnable [0x00007f36da6b2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000 nid=0x3a60 runnable [0x00007f36da7b3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800 nid=0x3a5f runnable [0x00007f36da8b4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800 nid=0x3a5e runnable [0x00007f36da9b5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition [0x00007f36dacb6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition [0x00007f36dadb7000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800 nid=0x3a43 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000 nid=0x3a42 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000 nid=0x3a41 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000 nid=0x3a40 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800 nid=0x3a3f runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d in Object.wait() [0x00007f36f51e5000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800 nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on condition [0x00007f3712fda000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a974ab8> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000 nid=0x3a34 runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800 nid=0x3a35 runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800 nid=0x3a36 runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000 nid=0x3a37 runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000 nid=0x3a3a runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44 waiting on condition
>>> 
>>> JNI global references: 486
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> And the driver:
>>> 
>>> 2015-03-27 02:20:47
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800 nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000 nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800 nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000 nid=0x41d0 waiting on condition [0x00007fd38feef000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0 tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000fd545eb8> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5 os_prio=0 tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition [0x00007fd3b8400000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000 nid=0x417f waiting on condition [0x00007fd398935000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800 nid=0x409c waiting on condition [0x00007fd3902f1000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000 nid=0x409b waiting on condition [0x00007fd3903f2000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800 nid=0x409a waiting on condition [0x00007fd3904f3000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800 nid=0x4098 waiting on condition [0x00007fd3905f4000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0 tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800 nid=0x4092 runnable [0x00007fd398b37000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e685f320> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800 nid=0x4091 runnable [0x00007fd398a36000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6871c30> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800 nid=0x4090 runnable [0x00007fd39903a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6873520> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0 tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0 tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e7f19070> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0 tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0 tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66fe220> (a java.util.concurrent.SynchronousQueue$TransferStack)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> 	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> 	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0 tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e740fa58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0 tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e740fce8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0 tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0 tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e737a0c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0 tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e737add8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0 tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0 tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>>> 
>>> "ResponseProcessor for block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable [0x00007fd39a941000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
>>> 	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>>> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>> 	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
>>> 	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
>>> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>>> 
>>> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000 nid=0x3fda waiting on condition [0x00007fd39aa42000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e58145f0> (a java.util.concurrent.Semaphore$NonfairSync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>>> 
>>> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000 nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>> 	at org.apache.spark.ContextCleaner.org <http://org.apache.spark.contextcleaner.org/>$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
>>> 	at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>>> 
>>> "DataStreamer for file /spark-logs/application_1427416187821_0004.inprogress block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait() [0x00007fd39ae44000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
>>> 	- locked <0x00000000e6994f30> (a java.util.LinkedList)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0 tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6995e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800 nid=0x3fb7 runnable [0x00007fd39b246000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6886b58> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Yarn application state monitor" #73 daemon prio=5 os_prio=0 tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>>> 
>>> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000 nid=0x3f77 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f waiting on condition [0x00007fd39b547000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e677ae98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "LeaseRenewer:hadoop@10.209.23.74:9000 <http://LeaseRenewer:hadoop@10.209.23.74:9000/>" #68 daemon prio=5 os_prio=0 tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e runnable [0x00007fd3b80ff000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "IPC Client (1489870911) connection to /10.209.23.74:9022 <http://10.209.23.74:9022/> from hadoop" #62 daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in Object.wait() [0x00007fd3b8501000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
>>> 	- locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
>>> 	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>>> 
>>> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0 tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9230> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
>>> 	at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
>>> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>>> 
>>> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in Object.wait() [0x00007fd3b8703000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e66e8190> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000 nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040 <http://SelectChannelConnector@0.0.0.0:4040/>" #54 daemon prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable [0x00007fd3b8fb6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>>> 	- locked <0x00000000e671a620> (a java.lang.Object)
>>> 	at org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0 tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>>> 	at org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0 tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e66609f0> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000 nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800 nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800 nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800 nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800 nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800 nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000 nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920 <http://SocketConnector@0.0.0.0:53920/>" #42 daemon prio=5 os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable [0x00007fd3ba0de000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
>>> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0 tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e6601270> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0 tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e6600ca8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0 tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e65380d8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "New I/O server boss #6" #36 daemon prio=5 os_prio=0 tid=0x00007fd39c04b800 nid=0x3df8 runnable [0x00007fd3ba6e2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800 nid=0x3df7 runnable [0x00007fd3ba7e3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800 nid=0x3df6 runnable [0x00007fd3ba8e4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000 nid=0x3df5 runnable [0x00007fd3ba9e5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800 nid=0x3df4 runnable [0x00007fd3baae6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000 nid=0x3df3 runnable [0x00007fd3babe7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5 os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition [0x00007fd3baef9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5 os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition [0x00007fd3baffa000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0 tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0 tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0 tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0 tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800 nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000 nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800 nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800 nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000 nid=0x3d72 waiting on condition [0x00007fd40c460000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000 nid=0x3d71 waiting on condition [0x00007fd40c561000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000 nid=0x3d70 waiting on condition [0x00007fd40c662000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131 <http://SocketConnector@0.0.0.0:53131/>" #14 daemon prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable [0x00007fd40c966000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
>>> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000 nid=0x3d67 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800 nid=0x3d66 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000 nid=0x3d65 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000 nid=0x3d64 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800 nid=0x3d63 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62 in Object.wait() [0x00007fd40f28e000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800 nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in Object.wait() [0x00007fd438220000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
>>> 	- locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
>>> 	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
>>> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
>>> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
>>> 	at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
>>> 	at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
>>> 	at org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
>>> 	at org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
>>> 	at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
>>> 	at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
>>> 	at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
>>> 	at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
>>> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
>>> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>>> 	at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
>>> 	at $line24.$read$$iwC$$iwC.<init>(<console>:35)
>>> 	at $line24.$read$$iwC.<init>(<console>:37)
>>> 	at $line24.$read.<init>(<console>:39)
>>> 	at $line24.$read$.<init>(<console>:43)
>>> 	at $line24.$read$.<clinit>(<console>)
>>> 	at $line24.$eval$.<init>(<console>:7)
>>> 	at $line24.$eval$.<clinit>(<console>)
>>> 	at $line24.$eval.$print(<console>)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:497)
>>> 	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>>> 	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>>> 	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>>> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>>> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>>> 	at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
>>> 	at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
>>> 	at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
>>> 	at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
>>> 	at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
>>> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>> 	at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>>> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>>> 	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
>>> 	at org.apache.spark.repl.Main$.main(Main.scala:31)
>>> 	at org.apache.spark.repl.Main.main(Main.scala)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:497)
>>> 	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>>> 	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>>> 	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>>> 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>>> 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>>> 
>>> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000 nid=0x3d5c runnable
>>> 
>>> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800 nid=0x3d5d runnable
>>> 
>>> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800 nid=0x3d5e runnable
>>> 
>>> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000 nid=0x3d5f runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68 waiting on condition
>>> 
>>> JNI global references: 420
>>> 
>>> 
>>> 
>>> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <michael@databricks.com <ma...@databricks.com>> wrote:
>>> Is it possible to jstack the executors and see where they are hanging?
>>> 
>>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jon.chase@gmail.com <ma...@gmail.com>> wrote:
>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB), executor memory 20GB, driver memory 10GB
>>> 
>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data spread out over roughly 2,000 Parquet files and my queries frequently hang. Simple queries like "select count(*) from ..." on the entire data set work ok.  Slightly more demanding ones with group by's and some aggregate functions (percentile_approx, avg, etc.) work ok as well, as long as I have some criteria in my where clause to keep the number of rows down.  
>>> 
>>> Once I hit some limit on query complexity and rows processed, my queries start to hang.  I've left them for up to an hour without seeing any progress.  No OOM's either - the job is just stuck.
>>> 
>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but with the same results: usually near the end of the tasks (like 780 of 800 complete), progress just stops:
>>> 
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in stage 1.0 (TID 1618) in 800 ms on ip-10-209-22-211.eu <http://ip-10-209-22-211.eu/>-west-1.compute.internal (748/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu <http://ip-10-105-12-41.eu/>-west-1.compute.internal (749/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu <http://ip-10-90-2-201.eu/>-west-1.compute.internal (750/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu <http://ip-10-90-2-201.eu/>-west-1.compute.internal (751/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu <http://ip-10-105-12-41.eu/>-west-1.compute.internal (752/800)
>>> 
>>> ^^^^^^^ this is where it stays forever
>>> 
>>> Looking at the Spark UI, several of the executors still list active tasks.  I do see that the Shuffle Read for executors that don't have any tasks remaining is around 100MB, whereas it's more like 10MB for the executors that still have tasks.  
>>> 
>>> The first stage, mapPartitions, always completes fine.  It's the second stage (takeOrdered), that hangs.
>>> 
>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also encountered it when using JSON files (instead of Parquet).
>>> 
>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do are having this issue.
>>> 
>>> 
>> 
> 
> 

Re: Spark SQL queries hang forever

[Michael Allman <mi...@videoamp.com>]

Hi Reynold,

Our build is based off of the 1.3 branch as of commit 6d3c4d8b04b2738a821dfcc3df55a5635b89e506. Unless your fix is on another branch it appears this problem still exists.

LMK if there's anything else I can help with.

Thanks,

Michael

> On Apr 16, 2015, at 11:44 AM, Reynold Xin <rx...@databricks.com> wrote:
> 
> Thanks for the information.
> 
> Adding the dev list.
> 
> Do you mind trying 1.3.1 to see if the issue is fixed?  I could be wrong but it migtht be related to https://issues.apache.org/jira/browse/SPARK-6578 <https://issues.apache.org/jira/browse/SPARK-6578>
> 
>  
> 
> On Thu, Apr 16, 2015 at 2:34 AM, Michael Allman <michael@videoamp.com <ma...@videoamp.com>> wrote:
> After many, many manual (and automated) trials using a spark sql query that *sometimes* triggers the hang I found the following:
> 
> When I set spark.shuffle.blockTransferService to netty (the default), I can sometimes reproduce the problem.
> When I set spark.shuffle.blockTransferService to nio (the Spark 1.1 default), I have not yet been able to reproduce the problem.
> 
> While far from a reliable conclusion, the workaround I'm going with for now is setting "spark.shuffle.blockTransferService nio" in spark-defaults.conf. One of the reasons I homed in on this setting as a possible trigger was the fact that its default value changed starting in Spark 1.2. We haven't experienced a similar problem in our current Spark 1.1 cluster, and we're skipping directly to Spark 1.3. Has anyone else tried this apparent workaround?
> 
> Cheers,
> 
> Michael
> 
>> On Apr 15, 2015, at 11:23 PM, Michael Allman <michael@videoamp.com <ma...@videoamp.com>> wrote:
>> 
>> FWIW I'm attempting an upgrade from Spark 1.1 to Spark 1.3 and I'm seeing this exact problem in ShuffleBlockFetcherIterator in the query takeOrdered stage. Sometimes a single task hangs, sometimes a couple dozen. The final stage has 200 tasks. I'm using the Spark standalone scheduler, parquet files. Here's the stack trace of the stuck thread:
>> 
>> sun.misc.Unsafe.park(Native Method)
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>> scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:389)
>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:362)
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:154)
>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:149)
>> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> org.apache.spark.scheduler.Task.run(Task.scala:64)
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> java.lang.Thread.run(Thread.java:745)
>> 
>> Anyone find a reliable fix/workaround? I'm going to try some things myself. If I find something that works I'll post back.
>> 
>> Cheers,
>> 
>> Michael
>> 
>> 
>>> On Mar 26, 2015, at 7:21 PM, Jon Chase <jon.chase@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> Here are the jstacks from a 3 worker cluster.  Two of the workers are sitting with 4 active tasks, the other has no active tasks.  Not sure if it makes a difference, but I only see an ExecutorLauncher instance via JPS on the worker that has no active tasks.  
>>> 
>>> Here's the first stack dump, this machine has 4 active tasks:
>>> 
>>> 2015-03-27 02:11:46
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000 nid=0x43a3 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5 os_prio=0 tid=0x00007f91d02fd000 nid=0x42ee waiting on condition [0x00007f91bad9a000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884fc0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000 nid=0x42e2 runnable [0x00007f91b4c54000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005898aa000> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800 nid=0x42e1 runnable [0x00007f91b5d56000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058ad5b5f8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000 nid=0x42e0 runnable [0x00007f91b5e57000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005898aa0c0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800 nid=0x42df runnable [0x00007f91b5f58000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058ad5b6b8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007f91d0773800 nid=0x427b waiting on condition [0x00007f91b6258000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cfda6800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0 tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition [0x00007f91b6b5a000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2 waiting on condition [0x00007f91b6c5b000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058af8e0b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef runnable [0x00007f91baf9c000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0 tid=0x00007f91d8118800 nid=0x41ee waiting on condition [0x00007f91bae9b000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000 nid=0x41ed runnable [0x00007f91baa97000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589224af0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0 tid=0x00007f91d0019000 nid=0x41ea waiting on condition [0x00007f91bac98000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006c8837750> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0 tid=0x00007f91d0016800 nid=0x41e9 waiting on condition [0x00007f91ba16d000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cc53bff0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0 tid=0x00007f91d0015000 nid=0x41e8 waiting on condition [0x00007f91b9f6b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006cfda69b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800 nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800 nid=0x41c6 runnable [0x00007f91bbac7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058984ee60> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0 tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition [0x00007f91bbbc8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005898f7290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0 tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058974cb00> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058974cb00> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0 tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x0000000589904038> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x0000000589904038> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0 tid=0x00007f91ed92d800 nid=0x41bf waiting on condition [0x00007f91bab98000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884e80> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0 tid=0x00007f91ed92c800 nid=0x41be waiting on condition [0x00007f91ba370000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O server boss #12" #39 daemon prio=5 os_prio=0 tid=0x0000000000a02000 nid=0x41bd runnable [0x00007f91b9564000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058992f188> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000 nid=0x41bc runnable [0x00007f91b9665000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589931218> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800 nid=0x41bb runnable [0x00007f91b9766000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800 nid=0x41ba runnable [0x00007f91b9867000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800 nid=0x41b9 runnable [0x00007f91b9968000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000 nid=0x41b8 runnable [0x00007f91b9c69000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon prio=5 os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition [0x00007f91b9d6a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon prio=5 os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition [0x00007f91b9e6b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589884f20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800 nid=0x419e runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000 nid=0x419d waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000 nid=0x419c waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000 nid=0x419b waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800 nid=0x419a runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f91ec101000 nid=0x4199 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198 in Object.wait() [0x00007f91d4254000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800 nid=0x4197 in Object.wait() [0x00007f91d4355000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on condition [0x00007f91f1fdf000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589a1b5f0> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000 nid=0x4191 runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800 nid=0x4192 runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800 nid=0x4193 runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000 nid=0x4194 runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000 nid=0x4195 runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f waiting on condition
>>> 
>>> JNI global references: 319
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Here's the next worker, also for active tasks:
>>> 
>>> 2015-03-27 02:12:54
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000 nid=0x44f1 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5 os_prio=0 tid=0x00007fcbe488b000 nid=0x4464 waiting on condition [0x00007fcbbf87f000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589510428> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0 tid=0x00007fcbe4028800 nid=0x445f waiting on condition [0x00007fcbbf97f000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006126cd358> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800 nid=0x4459 runnable [0x00007fcbc88c7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a6b8c98> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000 nid=0x4458 runnable [0x00007fcbc7897000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a6e5a18> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800 nid=0x4457 runnable [0x00007fcbc7998000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058aade110> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000 nid=0x4456 runnable [0x00007fcbc7b9a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058aade1d0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0 tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition [0x00007fcbc4886000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000615539dd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0 tid=0x00007fcbe4165000 nid=0x4453 waiting on condition [0x00007fcbca620000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000617bd4238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0 tid=0x00007fcbe4164000 nid=0x4452 waiting on condition [0x00007fcbc87c5000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000006126cd538> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>> 	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0 tid=0x00007fcbdcff1800 nid=0x436e waiting on condition [0x00007fcbc4485000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369 waiting on condition [0x00007fcbc4586000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058adabe70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366 runnable [0x00007fcbc7e9d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007fcbf9608800 nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800 nid=0x4364 runnable [0x00007fcbc8ac9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589116ae0> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000 nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000 nid=0x4340 runnable [0x00007fcbc94f3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058964c068> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition [0x00007fcbc97f6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a6fa058> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058928c438> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058928c438> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x000000058928c5a0> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007fcbdc29d000 nid=0x433a waiting on condition [0x00007fcbc85c4000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007fcbe4474800 nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a70a468> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007fcbdc29a000 nid=0x4337 runnable [0x00007fcbc6e8f000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000 nid=0x4336 runnable [0x00007fcbc6f90000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800 nid=0x4335 runnable [0x00007fcbc7091000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000 nid=0x4334 runnable [0x00007fcbc7392000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a796550> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000 nid=0x4333 runnable [0x00007fcbc7493000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000 nid=0x4332 runnable [0x00007fcbc7594000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition [0x00007fcbc7695000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition [0x00007fcbc7796000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005893c1820> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007fcbf995d800 nid=0x432a waiting on condition [0x00007fcbc7d9c000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800 nid=0x4318 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800 nid=0x4317 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000 nid=0x4316 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000 nid=0x4315 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000 nid=0x4314 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007fcbf8103800 nid=0x4313 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312 in Object.wait() [0x00007fcbe1be7000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800 nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on condition [0x00007fcbff9b9000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589585398> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000 nid=0x430b runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800 nid=0x430c runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800 nid=0x430d runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000 nid=0x430e runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000 nid=0x430f runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319 waiting on condition
>>> 
>>> JNI global references: 375
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> And the final worker, no active tasks:
>>> 
>>> 
>>> 2015-03-27 02:13:26
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000 nid=0x3c5b waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5 os_prio=0 tid=0x00007f36f072d000 nid=0x3b9a waiting on condition [0x00007f36d2e33000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005895676e8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000 nid=0x3b88 runnable [0x00007f36dafb9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a86a7d8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800 nid=0x3b87 runnable [0x00007f36d7138000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058abea6c8> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800 nid=0x3b86 runnable [0x00007f36daeb8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a86a898> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000 nid=0x3b85 runnable [0x00007f36d7e3b000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058abea788> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0 tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition [0x00007f36d7a39000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d waiting on condition [0x00007f36d7b3a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058aecd008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a runnable [0x00007f36db4be000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0 tid=0x00007f36f8025000 nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000 nid=0x3a96 runnable [0x00007f36dc0ea000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x0000000589202c58> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000 nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>> 
>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000 nid=0x3a6c runnable [0x00007f36dcb14000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058a825c18> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0 tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition [0x00007f36dce17000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005897334d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0 tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000005896b0d20> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0 tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	- waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000005896b0dc8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0 tid=0x00007f36f03a1000 nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>> 	at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0 tid=0x00007f370d9b2000 nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000005895675a8> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0 tid=0x00007f36f8713000 nid=0x3a63 runnable [0x00007f36da4b0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058970f538> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800 nid=0x3a62 runnable [0x00007f36da5b1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800 nid=0x3a61 runnable [0x00007f36da6b2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000 nid=0x3a60 runnable [0x00007f36da7b3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800 nid=0x3a5f runnable [0x00007f36da8b4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800 nid=0x3a5e runnable [0x00007f36da9b5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon prio=5 os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition [0x00007f36dacb6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon prio=5 os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition [0x00007f36dadb7000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x0000000589567648> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0 tid=0x00007f370d97c000 nid=0x3a56 waiting on condition [0x00007f36db3bd000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800 nid=0x3a43 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000 nid=0x3a42 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000 nid=0x3a41 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000 nid=0x3a40 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800 nid=0x3a3f runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f370c101800 nid=0x3a3e waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d in Object.wait() [0x00007f36f51e5000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800 nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on condition [0x00007f3712fda000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x000000058a974ab8> (a java.util.concurrent.CountDownLatch$Sync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>> 	at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>> 	at scala.concurrent.Await$.ready(package.scala:86)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>> 	at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:422)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>>> 
>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000 nid=0x3a34 runnable
>>> 
>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800 nid=0x3a35 runnable
>>> 
>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800 nid=0x3a36 runnable
>>> 
>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000 nid=0x3a37 runnable
>>> 
>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000 nid=0x3a3a runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44 waiting on condition
>>> 
>>> JNI global references: 486
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> And the driver:
>>> 
>>> 2015-03-27 02:20:47
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
>>> 
>>> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800 nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000 nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800 nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000 nid=0x41d0 waiting on condition [0x00007fd38feef000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0 tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition [0x00007fd38fff0000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000fd545eb8> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5 os_prio=0 tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition [0x00007fd3b8400000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000 nid=0x417f waiting on condition [0x00007fd398935000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-3" #107 daemon prio=5 os_prio=0 tid=0x00007fd3b003f800 nid=0x409c waiting on condition [0x00007fd3902f1000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-2" #106 daemon prio=5 os_prio=0 tid=0x00007fd3b003d000 nid=0x409b waiting on condition [0x00007fd3903f2000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-1" #105 daemon prio=5 os_prio=0 tid=0x00007fd3b003b800 nid=0x409a waiting on condition [0x00007fd3904f3000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "task-result-getter-0" #104 daemon prio=5 os_prio=0 tid=0x00007fd3b0030800 nid=0x4098 waiting on condition [0x00007fd3905f4000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0 tid=0x00007fd3cc00c800 nid=0x4093 waiting on condition [0x00007fd3936f8000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800 nid=0x4092 runnable [0x00007fd398b37000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e685f320> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800 nid=0x4091 runnable [0x00007fd398a36000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6871c30> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800 nid=0x4090 runnable [0x00007fd39903a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6873520> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0 tid=0x00007fd3e0011800 nid=0x4056 waiting on condition [0x00007fd398733000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0 tid=0x00007fd3dc001800 nid=0x4055 waiting on condition [0x00007fd398834000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e7f19070> (a scala.concurrent.forkjoin.ForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5 os_prio=0 tid=0x00007fd3a400d800 nid=0x4051 waiting on condition [0x00007fd398e38000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0 tid=0x00007fd432c05000 nid=0x404f waiting on condition [0x00007fd398f39000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66fe220> (a java.util.concurrent.SynchronousQueue$TransferStack)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> 	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> 	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0 tid=0x00007fd43297c800 nid=0x4042 waiting on condition [0x00007fd39973b000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e740fa58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0 tid=0x00007fd4329ac000 nid=0x4041 waiting on condition [0x00007fd39983c000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e740fce8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0 tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0 tid=0x00007fd4328e6800 nid=0x403b waiting on condition [0x00007fd399c3e000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e737a0c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>> 	at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0 tid=0x00007fd4328eb800 nid=0x403a waiting on condition [0x00007fd399d3f000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e737add8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0 tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0 tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>>> 
>>> "ResponseProcessor for block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable [0x00007fd39a941000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
>>> 	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>>> 	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>>> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>> 	at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>> 	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
>>> 	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
>>> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>>> 
>>> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000 nid=0x3fda waiting on condition [0x00007fd39aa42000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e58145f0> (a java.util.concurrent.Semaphore$NonfairSync)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> 	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>> 	at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>>> 
>>> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0 tid=0x00007fd43258c000 nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>> 	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>> 	at org.apache.spark.ContextCleaner.org <http://org.apache.spark.contextcleaner.org/>$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
>>> 	at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>>> 
>>> "DataStreamer for file /spark-logs/application_1427416187821_0004.inprogress block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84 daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in Object.wait() [0x00007fd39ae44000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
>>> 	- locked <0x00000000e6994f30> (a java.util.LinkedList)
>>> 
>>> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0 tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition [0x00007fd39af45000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6995e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800 nid=0x3fb7 runnable [0x00007fd39b246000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6886b58> (a io.netty.channel.nio.SelectedSelectionKeySet)
>>> 	- locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Yarn application state monitor" #73 daemon prio=5 os_prio=0 tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition [0x00007fd39b8fb000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>>> 
>>> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000 nid=0x3f77 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800 nid=0x3e6f waiting on condition [0x00007fd39b547000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e677ae98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "LeaseRenewer:hadoop@10.209.23.74:9000 <http://LeaseRenewer:hadoop@10.209.23.74:9000/>" #68 daemon prio=5 os_prio=0 tid=0x00007fd432896800 nid=0x3e6a waiting on condition [0x00007fd39b9fc000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>>> 	at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000 nid=0x3e5e runnable [0x00007fd3b80ff000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>> 	at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "IPC Client (1489870911) connection to /10.209.23.74:9022 <http://10.209.23.74:9022/> from hadoop" #62 daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in Object.wait() [0x00007fd3b8501000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
>>> 	- locked <0x00000000e66af0f8> (a org.apache.hadoop.ipc.Client$Connection)
>>> 	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>>> 
>>> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0 tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition [0x00007fd3b8602000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e66e9230> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> 	at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
>>> 	at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
>>> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>>> 
>>> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f in Object.wait() [0x00007fd3b8703000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e66e8190> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000 nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e67194f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040 <http://SelectChannelConnector@0.0.0.0:4040/>" #54 daemon prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable [0x00007fd3b8fb6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>>> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>>> 	- locked <0x00000000e671a620> (a java.lang.Object)
>>> 	at org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0 tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>>> 	at org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0 tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e66609f0> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000 nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800 nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800 nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800 nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800 nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800 nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000 nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e6539dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920 <http://SocketConnector@0.0.0.0:53920/>" #42 daemon prio=5 os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable [0x00007fd3ba0de000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
>>> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0 tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e6601270> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0 tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e6600ca8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0 tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.util.TimerThread.mainLoop(Timer.java:526)
>>> 	- locked <0x00000000e65380d8> (a java.util.TaskQueue)
>>> 	at java.util.TimerThread.run(Timer.java:505)
>>> 
>>> "New I/O server boss #6" #36 daemon prio=5 os_prio=0 tid=0x00007fd39c04b800 nid=0x3df8 runnable [0x00007fd3ba6e2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800 nid=0x3df7 runnable [0x00007fd3ba7e3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800 nid=0x3df6 runnable [0x00007fd3ba8e4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000 nid=0x3df5 runnable [0x00007fd3ba9e5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800 nid=0x3df4 runnable [0x00007fd3baae6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000 nid=0x3df3 runnable [0x00007fd3babe7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>> 	- locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
>>> 	- locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
>>> 	- locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
>>> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>> 	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>> 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>> 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5 os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition [0x00007fd3baef9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5 os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition [0x00007fd3baffa000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64bbb40> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5 os_prio=0 tid=0x00007fd431f99000 nid=0x3de0 waiting on condition [0x00007fd3bb0fb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5 os_prio=0 tid=0x00007fd431f96800 nid=0x3ddf waiting on condition [0x00007fd3bb1fc000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5 os_prio=0 tid=0x00007fd431f86000 nid=0x3dde waiting on condition [0x00007fd3bb2fd000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e64f49f8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0 tid=0x00007fd431f0e800 nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> 	at java.lang.Thread.sleep(Native Method)
>>> 	at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>> 	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800 nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000 nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800 nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800 nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000 nid=0x3d72 waiting on condition [0x00007fd40c460000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000 nid=0x3d71 waiting on condition [0x00007fd40c561000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000 nid=0x3d70 waiting on condition [0x00007fd40c662000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> 	at sun.misc.Unsafe.park(Native Method)
>>> 	- parking to wait for  <0x00000000e42808f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>> 	at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131 <http://SocketConnector@0.0.0.0:53131/>" #14 daemon prio=5 os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable [0x00007fd40c966000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>> 	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>> 	at java.net.ServerSocket.accept(ServerSocket.java:513)
>>> 	at org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>> 	at org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> 	at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> 
>>> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000 nid=0x3d67 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800 nid=0x3d66 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000 nid=0x3d65 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000 nid=0x3d64 waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800 nid=0x3d63 runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62 in Object.wait() [0x00007fd40f28e000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>> 	- locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
>>> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>> 
>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800 nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>> 	- locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in Object.wait() [0x00007fd438220000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> 	at java.lang.Object.wait(Native Method)
>>> 	at java.lang.Object.wait(Object.java:502)
>>> 	at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
>>> 	- locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
>>> 	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
>>> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
>>> 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
>>> 	at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
>>> 	at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
>>> 	at org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
>>> 	at org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
>>> 	at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
>>> 	at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
>>> 	at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
>>> 	at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
>>> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
>>> 	at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>>> 	at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>>> 	at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
>>> 	at $line24.$read$$iwC$$iwC.<init>(<console>:35)
>>> 	at $line24.$read$$iwC.<init>(<console>:37)
>>> 	at $line24.$read.<init>(<console>:39)
>>> 	at $line24.$read$.<init>(<console>:43)
>>> 	at $line24.$read$.<clinit>(<console>)
>>> 	at $line24.$eval$.<init>(<console>:7)
>>> 	at $line24.$eval$.<clinit>(<console>)
>>> 	at $line24.$eval.$print(<console>)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:497)
>>> 	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>>> 	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>>> 	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>>> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>>> 	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>>> 	at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
>>> 	at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
>>> 	at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
>>> 	at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
>>> 	at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
>>> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>> 	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>> 	at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>>> 	at org.apache.spark.repl.SparkILoop.org <http://org.apache.spark.repl.sparkiloop.org/>$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>>> 	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
>>> 	at org.apache.spark.repl.Main$.main(Main.scala:31)
>>> 	at org.apache.spark.repl.Main.main(Main.scala)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> 	at java.lang.reflect.Method.invoke(Method.java:497)
>>> 	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>>> 	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>>> 	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>>> 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>>> 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>> 
>>> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>>> 
>>> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000 nid=0x3d5c runnable
>>> 
>>> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800 nid=0x3d5d runnable
>>> 
>>> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800 nid=0x3d5e runnable
>>> 
>>> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000 nid=0x3d5f runnable
>>> 
>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68 waiting on condition
>>> 
>>> JNI global references: 420
>>> 
>>> 
>>> 
>>> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust <michael@databricks.com <ma...@databricks.com>> wrote:
>>> Is it possible to jstack the executors and see where they are hanging?
>>> 
>>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jon.chase@gmail.com <ma...@gmail.com>> wrote:
>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu, 30GB), executor memory 20GB, driver memory 10GB
>>> 
>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data spread out over roughly 2,000 Parquet files and my queries frequently hang. Simple queries like "select count(*) from ..." on the entire data set work ok.  Slightly more demanding ones with group by's and some aggregate functions (percentile_approx, avg, etc.) work ok as well, as long as I have some criteria in my where clause to keep the number of rows down.  
>>> 
>>> Once I hit some limit on query complexity and rows processed, my queries start to hang.  I've left them for up to an hour without seeing any progress.  No OOM's either - the job is just stuck.
>>> 
>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800, but with the same results: usually near the end of the tasks (like 780 of 800 complete), progress just stops:
>>> 
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0 in stage 1.0 (TID 1618) in 800 ms on ip-10-209-22-211.eu <http://ip-10-209-22-211.eu/>-west-1.compute.internal (748/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0 in stage 1.0 (TID 1623) in 622 ms on ip-10-105-12-41.eu <http://ip-10-105-12-41.eu/>-west-1.compute.internal (749/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0 in stage 1.0 (TID 1627) in 616 ms on ip-10-90-2-201.eu <http://ip-10-90-2-201.eu/>-west-1.compute.internal (750/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0 in stage 1.0 (TID 1629) in 611 ms on ip-10-90-2-201.eu <http://ip-10-90-2-201.eu/>-west-1.compute.internal (751/800)
>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0 in stage 1.0 (TID 1625) in 669 ms on ip-10-105-12-41.eu <http://ip-10-105-12-41.eu/>-west-1.compute.internal (752/800)
>>> 
>>> ^^^^^^^ this is where it stays forever
>>> 
>>> Looking at the Spark UI, several of the executors still list active tasks.  I do see that the Shuffle Read for executors that don't have any tasks remaining is around 100MB, whereas it's more like 10MB for the executors that still have tasks.  
>>> 
>>> The first stage, mapPartitions, always completes fine.  It's the second stage (takeOrdered), that hangs.
>>> 
>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also encountered it when using JSON files (instead of Parquet).
>>> 
>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do are having this issue.
>>> 
>>> 
>> 
> 
>