You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2018/01/08 19:57:00 UTC
[jira] [Commented] (SPARK-22318) spark stream Kafka hang at
JavaStreamingContext.start, no spark job create
[ https://issues.apache.org/jira/browse/SPARK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316907#comment-16316907 ]
Sean Owen commented on SPARK-22318:
-----------------------------------
I ran into this recently too. In my case, the problem was that there was only one Kafka broker running in the dev environment, but, {{offsets.topic.replication.factor}} was set to 2. Try setting it to 1. This isn't a good idea in production but may workaround the issue where you just need 1 broker for testing.
If that's your issue, then it's really a Kafka thing.
> spark stream Kafka hang at JavaStreamingContext.start, no spark job create
> --------------------------------------------------------------------------
>
> Key: SPARK-22318
> URL: https://issues.apache.org/jira/browse/SPARK-22318
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 2.1.0
> Environment: OS:Red Hat Enterprise Linux Server release 6.5
> JRE:Oracle 1.8.0.144-b01
> spark-streaming_2.11:2.1.0
> spark-streaming-kafka-0-10_2.11:2.1.0
> Reporter: iceriver322
>
> spark stream Kafka jar submitted by spark-submit to standalone spark cluster, and running well for a few days. But recently, we find that no new job generated for the stream, we tried to restart the job, and restart the cluster, the stream just stuck at JavaStreamingContext.start, and WAITING (on object monitor). Thread dump :
> 2017-10-19 16:44:23
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode):
> "Attach Listener" #82 daemon prio=9 os_prio=0 tid=0x00007f76f0002000 nid=0x3f80 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "SparkUI-JettyScheduler" #81 daemon prio=5 os_prio=0 tid=0x00007f76ac002800 nid=0x3d5c waiting on condition [0x00007f7693bfa000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fa19f940> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "shuffle-server-3-4" #35 daemon prio=5 os_prio=0 tid=0x00007f76a0041800 nid=0x3d34 runnable [0x00007f76911e5000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000f8ea3be8> (a io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000f8ee3600> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000f8ea3ae0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:760)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "shuffle-server-3-3" #34 daemon prio=5 os_prio=0 tid=0x00007f76a0040800 nid=0x3d33 runnable [0x00007f76912e6000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000fc2747c0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000fc2874c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fc2746c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:760)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "shuffle-server-3-2" #33 daemon prio=5 os_prio=0 tid=0x00007f76a003e800 nid=0x3d32 runnable [0x00007f76913e7000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000fb227370> (a io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000fb2296a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fb227278> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:760)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "ForkJoinPool-1-worker-5" #80 daemon prio=5 os_prio=0 tid=0x00007f76a0034800 nid=0x3d31 runnable [0x00007f76916e7000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000f8e8ed98> (a sun.nio.ch.Util$3)
> - locked <0x00000000f8e8ed88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000f8e7d008> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:454)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:277)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:134)
> at org.apache.kafka.clients.consumer.internals.Fetcher.listOffset(Fetcher.java:324)
> at org.apache.kafka.clients.consumer.internals.Fetcher.resetOffset(Fetcher.java:298)
> at org.apache.kafka.clients.consumer.internals.Fetcher.updateFetchPositions(Fetcher.java:174)
> at org.apache.kafka.clients.consumer.KafkaConsumer.updateFetchPositions(KafkaConsumer.java:1409)
> at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:983)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:938)
> at org.apache.spark.streaming.kafka010.DirectKafkaInputDStream.paranoidPoll(DirectKafkaInputDStream.scala:168)
> at org.apache.spark.streaming.kafka010.DirectKafkaInputDStream.start(DirectKafkaInputDStream.scala:244)
> at org.apache.spark.streaming.DStreamGraph$$anonfun$start$5.apply(DStreamGraph.scala:49)
> at org.apache.spark.streaming.DStreamGraph$$anonfun$start$5.apply(DStreamGraph.scala:49)
> at scala.collection.parallel.mutable.ParArray$ParArrayIterator.foreach_quick(ParArray.scala:143)
> at scala.collection.parallel.mutable.ParArray$ParArrayIterator.foreach(ParArray.scala:136)
> at scala.collection.parallel.ParIterableLike$Foreach.leaf(ParIterableLike.scala:972)
> at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49)
> at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
> at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
> at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51)
> at scala.collection.parallel.ParIterableLike$Foreach.tryLeaf(ParIterableLike.scala:969)
> at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:152)
> at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443)
> at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Locked ownable synchronizers:
> - None
> "JobGenerator" #79 daemon prio=5 os_prio=0 tid=0x00007f76a0007800 nid=0x3d30 waiting on condition [0x00007f76917e9000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe48b8d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
> Locked ownable synchronizers:
> - None
> "JobScheduler" #78 daemon prio=5 os_prio=0 tid=0x00007f76a0004800 nid=0x3d2f waiting on condition [0x00007f76918ea000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe48cb98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
> Locked ownable synchronizers:
> - None
> "streaming-start" #77 daemon prio=5 os_prio=0 tid=0x00007f77323a1000 nid=0x3d2e in Object.wait() [0x00007f76919ea000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at scala.concurrent.forkjoin.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:295)
> - locked <0x00000000fa037d50> (a scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask)
> at scala.concurrent.forkjoin.ForkJoinTask.doJoin(ForkJoinTask.java:341)
> at scala.concurrent.forkjoin.ForkJoinTask.join(ForkJoinTask.java:673)
> at scala.collection.parallel.ForkJoinTasks$WrappedTask$class.sync(Tasks.scala:378)
> at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:443)
> at scala.collection.parallel.ForkJoinTasks$class.executeAndWaitResult(Tasks.scala:426)
> at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:56)
> at scala.collection.parallel.ExecutionContextTasks$class.executeAndWaitResult(Tasks.scala:558)
> at scala.collection.parallel.ExecutionContextTaskSupport.executeAndWaitResult(TaskSupport.scala:80)
> at scala.collection.parallel.ParIterableLike$class.foreach(ParIterableLike.scala:463)
> at scala.collection.parallel.mutable.ParArray.foreach(ParArray.scala:56)
> at org.apache.spark.streaming.DStreamGraph.start(DStreamGraph.scala:49)
> - locked <0x00000000fa0380d0> (a org.apache.spark.streaming.DStreamGraph)
> at org.apache.spark.streaming.scheduler.JobGenerator.startFirstTime(JobGenerator.scala:194)
> at org.apache.spark.streaming.scheduler.JobGenerator.start(JobGenerator.scala:100)
> - locked <0x00000000fe48b4d0> (a org.apache.spark.streaming.scheduler.JobGenerator)
> at org.apache.spark.streaming.scheduler.JobScheduler.start(JobScheduler.scala:102)
> - locked <0x00000000fe48b170> (a org.apache.spark.streaming.scheduler.JobScheduler)
> at org.apache.spark.streaming.StreamingContext$$anonfun$liftedTree1$1$1.apply$mcV$sp(StreamingContext.scala:583)
> at org.apache.spark.streaming.StreamingContext$$anonfun$liftedTree1$1$1.apply(StreamingContext.scala:578)
> at org.apache.spark.streaming.StreamingContext$$anonfun$liftedTree1$1$1.apply(StreamingContext.scala:578)
> at org.apache.spark.util.ThreadUtils$$anon$2.run(ThreadUtils.scala:126)
> Locked ownable synchronizers:
> - None
> "SparkListenerBus" #21 daemon prio=5 os_prio=0 tid=0x00007f7732291800 nid=0x3d2d waiting on condition [0x00007f7691cec000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000081c9be70> (a java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:80)
> at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
> at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
> at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1245)
> at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77)
> Locked ownable synchronizers:
> - None
> "context-cleaner-periodic-gc" #75 daemon prio=5 os_prio=0 tid=0x00007f773228b800 nid=0x3d2c waiting on condition [0x00007f7691ded000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe467df8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "Spark Context Cleaner" #74 daemon prio=5 os_prio=0 tid=0x00007f773228a000 nid=0x3d2b in Object.wait() [0x00007f7691eee000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000fe4675d0> (a java.lang.ref.ReferenceQueue$Lock)
> at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:175)
> at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1245)
> at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:172)
> at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:67)
> Locked ownable synchronizers:
> - None
> "shuffle-server-6-1" #70 daemon prio=5 os_prio=0 tid=0x00007f77321b5800 nid=0x3d2a runnable [0x00007f7691fef000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000fa182e28> (a io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000fa1b3938> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fa182d90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:760)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "threadDeathWatcher-4-1" #65 daemon prio=1 os_prio=0 tid=0x00007f7704019800 nid=0x3d29 waiting on condition [0x00007f76932f1000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:150)
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "shuffle-client-1-3" #30 daemon prio=5 os_prio=0 tid=0x00007f76fc006800 nid=0x3d28 runnable [0x00007f76933f2000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000fd326748> (a io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000fd328838> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fd3266a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:760)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "shuffle-client-1-2" #29 daemon prio=5 os_prio=0 tid=0x00007f770801a800 nid=0x3d27 runnable [0x00007f76934f3000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000fe49d458> (a io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000fe4b02d0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fe49d360> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:760)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "shuffle-client-1-1" #28 daemon prio=5 os_prio=0 tid=0x00007f7700005000 nid=0x3d26 runnable [0x00007f76935f4000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000fe4b24a0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000fe4b4570> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fe4b23a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:760)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "appclient-registration-retry-thread" #61 daemon prio=5 os_prio=0 tid=0x00007f76a800f000 nid=0x3d22 waiting on condition [0x00007f76939f8000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe4e0e28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "driver-revive-thread" #57 daemon prio=5 os_prio=0 tid=0x00007f76b0004000 nid=0x3d1e waiting on condition [0x00007f7693dfc000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe4e0450> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "dag-scheduler-event-loop" #56 daemon prio=5 os_prio=0 tid=0x00007f77321a5800 nid=0x3d1d waiting on condition [0x00007f7693efd000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe38ff08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
> Locked ownable synchronizers:
> - None
> "netty-rpc-env-timeout" #55 daemon prio=5 os_prio=0 tid=0x00007f76d0012000 nid=0x3d1c waiting on condition [0x00007f7693ffe000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe4b5268> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "Timer-1" #54 daemon prio=5 os_prio=0 tid=0x00007f773216b800 nid=0x3d1b in Object.wait() [0x00007f76dc1f3000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.util.TimerThread.mainLoop(Timer.java:526)
> - locked <0x00000000fb221ce8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> Locked ownable synchronizers:
> - None
> "heartbeat-receiver-event-loop-thread" #53 daemon prio=5 os_prio=0 tid=0x00007f76a4006000 nid=0x3d1a waiting on condition [0x00007f76dc4f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe46ae98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "SparkUI-52" #52 daemon prio=5 os_prio=0 tid=0x00007f7732129800 nid=0x3d19 waiting on condition [0x00007f76dc5f5000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe457aa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "SparkUI-51" #51 daemon prio=5 os_prio=0 tid=0x00007f7732127800 nid=0x3d18 waiting on condition [0x00007f76dc6f6000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe457aa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "SparkUI-50" #50 daemon prio=5 os_prio=0 tid=0x00007f7732125800 nid=0x3d17 waiting on condition [0x00007f76dc7f7000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe457aa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "SparkUI-49" #49 daemon prio=5 os_prio=0 tid=0x00007f7732123800 nid=0x3d16 waiting on condition [0x00007f76dc8f8000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe457aa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "SparkUI-48-acceptor-0@5bebf516-ServerConnector@2f9e7827{HTTP/1.1}{0.0.0.0:4040}" #48 daemon prio=5 os_prio=0 tid=0x00007f7732122000 nid=0x3d15 runnable [0x00007f76dc9f9000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> - locked <0x00000000fa1a15d8> (a java.lang.Object)
> at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:377)
> at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "SparkUI-47-selector-ServerConnectorManager@4e80bfcc/1" #47 daemon prio=5 os_prio=0 tid=0x00007f7732120000 nid=0x3d14 runnable [0x00007f76dcafa000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000fa1a23c0> (a sun.nio.ch.Util$3)
> - locked <0x00000000fa1a23b0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fa1a2288> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
> at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
> at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "SparkUI-46-selector-ServerConnectorManager@4e80bfcc/0" #46 daemon prio=5 os_prio=0 tid=0x00007f773211e000 nid=0x3d13 runnable [0x00007f76dcbfb000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000fa1a0b98> (a sun.nio.ch.Util$3)
> - locked <0x00000000fa1a0b88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fa1a07d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
> at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
> at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "SparkUI-45" #45 daemon prio=5 os_prio=0 tid=0x00007f7732116000 nid=0x3d12 waiting on condition [0x00007f76dccfc000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fe457aa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "map-output-dispatcher-7" #43 daemon prio=5 os_prio=0 tid=0x00007f7732021000 nid=0x3d0f waiting on condition [0x00007f76dcffd000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f8ea1380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:340)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000f8ea2450> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "map-output-dispatcher-6" #42 daemon prio=5 os_prio=0 tid=0x00007f773201f000 nid=0x3d0e waiting on condition [0x00007f76f415c000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f8ea1380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:340)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000f8ea2148> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "map-output-dispatcher-5" #41 daemon prio=5 os_prio=0 tid=0x00007f773201d000 nid=0x3d0d waiting on condition [0x00007f76f425d000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f8ea1380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:340)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000f8ea10c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "map-output-dispatcher-4" #40 daemon prio=5 os_prio=0 tid=0x00007f773201b000 nid=0x3d0c waiting on condition [0x00007f76f435e000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f8ea1380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:340)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000f8ea1e60> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "map-output-dispatcher-3" #39 daemon prio=5 os_prio=0 tid=0x00007f7732019000 nid=0x3d0b waiting on condition [0x00007f76f445f000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f8ea1380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:340)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000f8ea1890> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "map-output-dispatcher-2" #38 daemon prio=5 os_prio=0 tid=0x00007f7732017000 nid=0x3d0a waiting on condition [0x00007f76f4560000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f8ea1380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:340)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000f8ea15a8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "map-output-dispatcher-1" #37 daemon prio=5 os_prio=0 tid=0x00007f7732015000 nid=0x3d09 waiting on condition [0x00007f76f4661000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f8ea1380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:340)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000f8ea1b78> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "map-output-dispatcher-0" #36 daemon prio=5 os_prio=0 tid=0x00007f7732014000 nid=0x3d08 waiting on condition [0x00007f76f4762000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f8ea1380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:340)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000f8ea0e50> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "shuffle-server-3-1" #32 daemon prio=5 os_prio=0 tid=0x00007f7731fea800 nid=0x3d07 runnable [0x00007f76f4863000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000fc26eeb8> (a io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x00000000fc271470> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fc26edb0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:760)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "dispatcher-event-loop-3" #27 daemon prio=5 os_prio=0 tid=0x00007f7731e6d800 nid=0x3cf7 waiting on condition [0x00007f76f4b64000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000081bae110> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x0000000081bad878> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "dispatcher-event-loop-2" #26 daemon prio=5 os_prio=0 tid=0x00007f7731e65800 nid=0x3cf6 waiting on condition [0x00007f76f4c65000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000081bae110> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x0000000081badaf0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "dispatcher-event-loop-1" #25 daemon prio=5 os_prio=0 tid=0x00007f7731e63800 nid=0x3cf5 waiting on condition [0x00007f76f4d66000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000081bae110> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x0000000081bae600> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "dispatcher-event-loop-0" #24 daemon prio=5 os_prio=0 tid=0x00007f7731e62800 nid=0x3cf4 waiting on condition [0x00007f76f5268000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000081bae110> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x0000000081bae318> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "Thread-5" #19 daemon prio=5 os_prio=0 tid=0x00007f7731799800 nid=0x3cf0 waiting on condition [0x00007f76f5569000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at com.ccb.iomp.appmon.redis.config.HystrixConfig$1.run(HystrixConfig.java:76)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "kafka-producer-network-thread | producer-1" #18 daemon prio=5 os_prio=0 tid=0x00007f773187a800 nid=0x3ce5 runnable [0x00007f76f5d6b000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000081bb6f80> (a sun.nio.ch.Util$3)
> - locked <0x0000000081bb6f70> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000081bb6e48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:454)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:277)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:229)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "commons-pool-EvictionTimer" #17 daemon prio=5 os_prio=0 tid=0x00007f7731bb2800 nid=0x3cdf in Object.wait() [0x00007f76f5c6a000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00000000813a9c20> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> Locked ownable synchronizers:
> - None
> "pool-1-thread-1" #15 prio=5 os_prio=0 tid=0x00007f76c8001800 nid=0x3cdd waiting on condition [0x00007f76f606c000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000813d3350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "zkCnxnScheduler-1-EventThread" #14 daemon prio=5 os_prio=0 tid=0x00007f76d0027800 nid=0x3cdc waiting on condition [0x00007f76f616d000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000813d3b10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
> Locked ownable synchronizers:
> - None
> "zkCnxnScheduler-1-SendThread(11.168.0.59:2181)" #13 daemon prio=5 os_prio=0 tid=0x00007f76d002a800 nid=0x3cdb runnable [0x00007f771018e000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000813fb098> (a sun.nio.ch.Util$3)
> - locked <0x00000000813fb088> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000813fac20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> Locked ownable synchronizers:
> - None
> "zkCnxnScheduler-1" #12 prio=5 os_prio=0 tid=0x00007f7731b84000 nid=0x3cda waiting on condition [0x00007f771028f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000000008140be40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "Thread-2" #11 prio=5 os_prio=0 tid=0x00007f7731b5c000 nid=0x3cd9 waiting on condition [0x00007f7710590000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at com.ccb.iomp.appmon.dataaccess.es.PmTaskMonitor.run(PmTaskMonitor.java:161)
> Locked ownable synchronizers:
> - None
> "Timer-0" #10 daemon prio=5 os_prio=0 tid=0x00007f7730eca000 nid=0x3cba in Object.wait() [0x00007f77110ec000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00000000804cefa8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> Locked ownable synchronizers:
> - None
> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f77300cc000 nid=0x3cb3 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f77300c0800 nid=0x3cb2 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f77300bf000 nid=0x3cb1 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f77300bc000 nid=0x3cb0 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f77300ba800 nid=0x3caf runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f7730087800 nid=0x3cae in Object.wait() [0x00007f772046f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x0000000080177188> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> Locked ownable synchronizers:
> - None
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f7730083000 nid=0x3cad in Object.wait() [0x00007f7720570000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0x0000000080184658> (a java.lang.ref.Reference$Lock)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> Locked ownable synchronizers:
> - None
> "main" #1 prio=5 os_prio=0 tid=0x00007f773000d800 nid=0x3ca7 in Object.wait() [0x00007f7736da7000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1252)
> - locked <0x00000000fe463ad0> (a org.apache.spark.util.ThreadUtils$$anon$2)
> at java.lang.Thread.join(Thread.java:1326)
> at org.apache.spark.util.ThreadUtils$.runInNewThread(ThreadUtils.scala:135)
> at org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:578)
> at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:572)
> - locked <0x0000000081c9ea00> (a java.lang.Object)
> - locked <0x0000000081f19c80> (a org.apache.spark.streaming.StreamingContext)
> at org.apache.spark.streaming.api.java.JavaStreamingContext.start(JavaStreamingContext.scala:556)
> at com.ccb.iomp.appmon.analysis.statistic.processor.StatisticProcessor.start(StatisticProcessor.java:780)
> at com.ccb.iomp.appmon.analysis.statistic.TranLogStatisticApp.main(TranLogStatisticApp.java:49)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Locked ownable synchronizers:
> - None
> "VM Thread" os_prio=0 tid=0x00007f773007b800 nid=0x3cac runnable
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f7730022800 nid=0x3ca8 runnable
> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f7730024800 nid=0x3ca9 runnable
> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f7730026000 nid=0x3caa runnable
> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f7730028000 nid=0x3cab runnable
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f77300cf000 nid=0x3cb4 waiting on condition
> JNI global references: 302
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org