You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ajith S (Jira)" <ji...@apache.org> on 2020/01/12 06:06:00 UTC
[jira] [Commented] (SPARK-30488) Deadlock between
block-manager-slave-async-thread-pool and spark context cleaner
[ https://issues.apache.org/jira/browse/SPARK-30488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013651#comment-17013651 ]
Ajith S commented on SPARK-30488:
---------------------------------
As from log, i see `sbt` classes in the deadlock threads,this is related to internal classloaders in sbt which was fixed in sbt 1.3.3 by marking classloaders as parallel capable. [https://github.com/sbt/sbt/pull/5131] also a similar issue '[https://github.com/sbt/sbt/issues/5116]'
[~rohit21agrawal] Thanks for reporting this. Some questions, can you also please mention how the sparkcontext was created.?
> Deadlock between block-manager-slave-async-thread-pool and spark context cleaner
> --------------------------------------------------------------------------------
>
> Key: SPARK-30488
> URL: https://issues.apache.org/jira/browse/SPARK-30488
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.3
> Reporter: Rohit Agrawal
> Priority: Major
>
> Deadlock happens while cleaning up the spark context. Here is the full thread dump:
>
>
> 2020-01-10T20:13:16.2884057Z Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.221-b11 mixed mode):
> 2020-01-10T20:13:16.2884392Z
> 2020-01-10T20:13:16.2884660Z "SIGINT handler" #488 daemon prio=9 os_prio=2 tid=0x00000000111fa000 nid=0x4794 waiting for monitor entry [0x000000001c86e000]
> 2020-01-10T20:13:16.2884807Z java.lang.Thread.State: BLOCKED (on object monitor)
> 2020-01-10T20:13:16.2884879Z at java.lang.Shutdown.exit(Shutdown.java:212)
> 2020-01-10T20:13:16.2885693Z - waiting to lock <0x00000000c0155de0> (a java.lang.Class for java.lang.Shutdown)
> 2020-01-10T20:13:16.2885840Z at java.lang.Terminator$1.handle(Terminator.java:52)
> 2020-01-10T20:13:16.2885965Z at sun.misc.Signal$1.run(Signal.java:212)
> 2020-01-10T20:13:16.2886329Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2886430Z
> 2020-01-10T20:13:16.2886752Z "Thread-3" #108 prio=5 os_prio=0 tid=0x00000000111f7800 nid=0x48cc waiting for monitor entry [0x000000002c33f000]
> 2020-01-10T20:13:16.2886881Z java.lang.Thread.State: BLOCKED (on object monitor)
> 2020-01-10T20:13:16.2886999Z at org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:273)
> 2020-01-10T20:13:16.2887107Z at org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:121)
> 2020-01-10T20:13:16.2887212Z at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95)
> 2020-01-10T20:13:16.2887421Z
> 2020-01-10T20:13:16.2887798Z "block-manager-slave-async-thread-pool-81" #486 daemon prio=5 os_prio=0 tid=0x00000000111fe800 nid=0x2e34 waiting for monitor entry [0x000000002bf3d000]
> 2020-01-10T20:13:16.2889192Z java.lang.Thread.State: BLOCKED (on object monitor)
> 2020-01-10T20:13:16.2889305Z at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> 2020-01-10T20:13:16.2889405Z - waiting to lock <0x00000000c1f359f0> (a sbt.internal.LayeredClassLoader)
> 2020-01-10T20:13:16.2889482Z at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
> 2020-01-10T20:13:16.2889582Z - locked <0x00000000ca33e4c8> (a sbt.internal.ManagedClassLoader$ZombieClassLoader)
> 2020-01-10T20:13:16.2889659Z at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 2020-01-10T20:13:16.2890881Z at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply$mcZ$sp(BlockManagerSlaveEndpoint.scala:58)
> 2020-01-10T20:13:16.2891006Z at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply(BlockManagerSlaveEndpoint.scala:57)
> 2020-01-10T20:13:16.2891142Z at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply(BlockManagerSlaveEndpoint.scala:57)
> 2020-01-10T20:13:16.2891260Z at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$1.apply(BlockManagerSlaveEndpoint.scala:86)
> 2020-01-10T20:13:16.2891375Z at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> 2020-01-10T20:13:16.2891624Z at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> 2020-01-10T20:13:16.2891737Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.2891833Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2891925Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2891967Z
> 2020-01-10T20:13:16.2892066Z "pool-31-thread-16" #335 prio=5 os_prio=0 tid=0x00000000153b2000 nid=0x1aac waiting on condition [0x000000004b2ff000]
> 2020-01-10T20:13:16.2892147Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2892241Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2892328Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2892437Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2892530Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2892649Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2892753Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2892865Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2892969Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2893060Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2893150Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2893192Z
> 2020-01-10T20:13:16.2893271Z "pool-31-thread-15" #334 prio=5 os_prio=0 tid=0x000000001a699000 nid=0x1110 waiting for monitor entry [0x000000004aefe000]
> 2020-01-10T20:13:16.2893375Z java.lang.Thread.State: BLOCKED (on object monitor)
> 2020-01-10T20:13:16.2893457Z at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:30)
> 2020-01-10T20:13:16.2893574Z at com.microsoft.azure.eventhubs.impl.MessageSender.onClose(MessageSender.java:424)
> 2020-01-10T20:13:16.2893849Z at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:70)
> 2020-01-10T20:13:16.2894172Z at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.handleRemoteLinkClosed(BaseLinkHandler.java:106)
> 2020-01-10T20:13:16.2894278Z at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:48)
> 2020-01-10T20:13:16.2894469Z at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
> 2020-01-10T20:13:16.2894565Z at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
> 2020-01-10T20:13:16.2894645Z at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
> 2020-01-10T20:13:16.2894745Z at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
> 2020-01-10T20:13:16.2894827Z at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:512)
> 2020-01-10T20:13:16.2894997Z at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2020-01-10T20:13:16.2895073Z at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2020-01-10T20:13:16.2895178Z at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> 2020-01-10T20:13:16.2895271Z at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> 2020-01-10T20:13:16.2895389Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.2895469Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2895555Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2895594Z
> 2020-01-10T20:13:16.2895691Z "pool-31-thread-14" #333 prio=5 os_prio=0 tid=0x000000001a697800 nid=0x31d4 waiting on condition [0x000000004aafe000]
> 2020-01-10T20:13:16.2895764Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2895855Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2895936Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2896037Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2896122Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2896238Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2896329Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2896433Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2896532Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2896613Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2896708Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2896748Z
> 2020-01-10T20:13:16.2896819Z "pool-31-thread-13" #332 prio=5 os_prio=0 tid=0x000000001a69e800 nid=0x4fe8 waiting on condition [0x000000004a6ff000]
> 2020-01-10T20:13:16.2896914Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2897080Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2897183Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2897265Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2897370Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2897461Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2897576Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2897659Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2897759Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2897836Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2898012Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2898050Z
> 2020-01-10T20:13:16.2898140Z "pool-31-thread-12" #331 prio=5 os_prio=0 tid=0x000000001a697000 nid=0x3a7c waiting on condition [0x000000004a2fe000]
> 2020-01-10T20:13:16.2898212Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2898294Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2898376Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2898536Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2898795Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2898911Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2899003Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2899115Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2899370Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2899474Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2899719Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2899787Z
> 2020-01-10T20:13:16.2899866Z "pool-31-thread-11" #330 prio=5 os_prio=0 tid=0x000000001a69d800 nid=0x5e48 waiting on condition [0x0000000049eff000]
> 2020-01-10T20:13:16.2899975Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2900043Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2900155Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2900253Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2900369Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2900469Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2900587Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2900679Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2900794Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2900900Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2900975Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2901016Z
> 2020-01-10T20:13:16.2901118Z "pool-31-thread-10" #329 prio=5 os_prio=0 tid=0x000000001a69d000 nid=0x1e48 waiting on condition [0x0000000049aff000]
> 2020-01-10T20:13:16.2901208Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2901297Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2901386Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2901496Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2901589Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2901720Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2901840Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2901932Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2902117Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2902205Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2902300Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2902343Z
> 2020-01-10T20:13:16.2902422Z "pool-31-thread-9" #328 prio=5 os_prio=0 tid=0x000000001a69c000 nid=0x558c waiting on condition [0x00000000496fe000]
> 2020-01-10T20:13:16.2902524Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2902614Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2902776Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2903046Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2903138Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2903265Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2903362Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2903473Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2903557Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2903664Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2903744Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2903804Z
> 2020-01-10T20:13:16.2903881Z "pool-31-thread-8" #327 prio=5 os_prio=0 tid=0x000000001a68f800 nid=0x12a8 waiting on condition [0x00000000492ff000]
> 2020-01-10T20:13:16.2903980Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2904046Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2904157Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2904400Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2904510Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2904604Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2904715Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2904807Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2905125Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2905202Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2905296Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2905334Z
> 2020-01-10T20:13:16.2905425Z "pool-31-thread-7" #326 prio=5 os_prio=0 tid=0x000000001a696000 nid=0x2e08 waiting on condition [0x0000000048eff000]
> 2020-01-10T20:13:16.2905498Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2905582Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2905662Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2905762Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2905853Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2905964Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2906052Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2906230Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2906333Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2906412Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2906498Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2906537Z
> 2020-01-10T20:13:16.2906608Z "pool-31-thread-6" #325 prio=5 os_prio=0 tid=0x000000001a694000 nid=0x3d24 waiting on condition [0x0000000048aff000]
> 2020-01-10T20:13:16.2906759Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2906822Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2906922Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2907003Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2907115Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2907224Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2907313Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2907414Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2907493Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2907599Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2907667Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2907725Z
> 2020-01-10T20:13:16.2907795Z "pool-31-thread-5" #324 prio=5 os_prio=0 tid=0x000000001a691000 nid=0x3c58 waiting on condition [0x00000000486fe000]
> 2020-01-10T20:13:16.2907892Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2907954Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2908054Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2908135Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2908237Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2908329Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2908443Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2908526Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2908628Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2908715Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2908800Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2908838Z
> 2020-01-10T20:13:16.2908930Z "pool-31-thread-4" #323 prio=5 os_prio=0 tid=0x000000001a692800 nid=0x360c waiting on condition [0x00000000482ff000]
> 2020-01-10T20:13:16.2909003Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2909085Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2909166Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2909277Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2909362Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2909473Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2909634Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2909736Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2909990Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2910094Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2910162Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2910284Z
> 2020-01-10T20:13:16.2910359Z "pool-31-thread-3" #322 prio=5 os_prio=0 tid=0x000000001a693000 nid=0x1554 waiting on condition [0x0000000047efe000]
> 2020-01-10T20:13:16.2910642Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2910882Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2910991Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2911088Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2911201Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2911302Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2911420Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2911536Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2911623Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2911730Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2911806Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2911855Z
> 2020-01-10T20:13:16.2911955Z "pool-31-thread-2" #321 prio=5 os_prio=0 tid=0x000000001a12f000 nid=0x4be4 waiting on condition [0x0000000047afe000]
> 2020-01-10T20:13:16.2912036Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2912128Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2912217Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2912326Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2912439Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2912548Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2912666Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2912758Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2912871Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2912958Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2913053Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2913224Z
> 2020-01-10T20:13:16.2913327Z "pool-31-thread-1" #320 prio=5 os_prio=0 tid=0x000000001a130800 nid=0x567c waiting on condition [0x00000000476ff000]
> 2020-01-10T20:13:16.2913408Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2913504Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2913593Z - parking to wait for <0x00000000c9cad078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2913702Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2913796Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2913994Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2914094Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2914206Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2914313Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2914465Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2914560Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2914602Z
> 2020-01-10T20:13:16.2914840Z "task-result-getter-3" #170 daemon prio=5 os_prio=0 tid=0x000000001a695800 nid=0x5ac8 waiting on condition [0x000000003eb3e000]
> 2020-01-10T20:13:16.2914945Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2915019Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2915287Z - parking to wait for <0x00000000c3ce7cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2915390Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2915478Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2915585Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.2915674Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2915773Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2916071Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2916159Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2916203Z
> 2020-01-10T20:13:16.2916277Z "task-result-getter-2" #169 daemon prio=5 os_prio=0 tid=0x000000001a694800 nid=0x62f0 waiting on condition [0x000000003e73f000]
> 2020-01-10T20:13:16.2916372Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2916455Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2916535Z - parking to wait for <0x00000000c3ce7cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2916635Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2916720Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2916829Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.2916907Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2917006Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2917090Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2917177Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2917214Z
> 2020-01-10T20:13:16.2917305Z "task-result-getter-1" #168 daemon prio=5 os_prio=0 tid=0x000000001a690000 nid=0x5c60 waiting on condition [0x000000003e33f000]
> 2020-01-10T20:13:16.2917381Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2917462Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2917540Z - parking to wait for <0x00000000c3ce7cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2917650Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2917734Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2917835Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.2917985Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2918082Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2918160Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2918247Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2918284Z
> 2020-01-10T20:13:16.2918374Z "ABFS-prefetch-7" #162 daemon prio=5 os_prio=0 tid=0x000000001f6ae000 nid=0x63e0 in Object.wait() [0x000000003b33e000]
> 2020-01-10T20:13:16.2918509Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.2941181Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2941508Z at java.lang.Object.wait(Object.java:502)
> 2020-01-10T20:13:16.2941599Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338)
> 2020-01-10T20:13:16.2941688Z - locked <0x00000000c583a5e8> (a org.apache.hadoop.fs.azurebfs.services.ReadBufferManager)
> 2020-01-10T20:13:16.2941799Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56)
> 2020-01-10T20:13:16.2941870Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2941930Z
> 2020-01-10T20:13:16.2942007Z "ABFS-prefetch-6" #161 daemon prio=5 os_prio=0 tid=0x000000001f6b0800 nid=0x4f28 in Object.wait() [0x000000003af3e000]
> 2020-01-10T20:13:16.2942093Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.2942157Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2942236Z at java.lang.Object.wait(Object.java:502)
> 2020-01-10T20:13:16.2942315Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338)
> 2020-01-10T20:13:16.2942406Z - locked <0x00000000c583a5e8> (a org.apache.hadoop.fs.azurebfs.services.ReadBufferManager)
> 2020-01-10T20:13:16.2942488Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56)
> 2020-01-10T20:13:16.2942571Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2942610Z
> 2020-01-10T20:13:16.2942683Z "ABFS-prefetch-5" #160 daemon prio=5 os_prio=0 tid=0x000000001f6b2800 nid=0x254c in Object.wait() [0x000000003ab3f000]
> 2020-01-10T20:13:16.2942770Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.2942839Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2942902Z at java.lang.Object.wait(Object.java:502)
> 2020-01-10T20:13:16.2943155Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338)
> 2020-01-10T20:13:16.2943252Z - locked <0x00000000c583a5e8> (a org.apache.hadoop.fs.azurebfs.services.ReadBufferManager)
> 2020-01-10T20:13:16.2943344Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56)
> 2020-01-10T20:13:16.2943416Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2943455Z
> 2020-01-10T20:13:16.2943542Z "ABFS-prefetch-4" #159 daemon prio=5 os_prio=0 tid=0x000000001f6b2000 nid=0x5c44 in Object.wait() [0x000000003a73e000]
> 2020-01-10T20:13:16.2943622Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.2943695Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2943758Z at java.lang.Object.wait(Object.java:502)
> 2020-01-10T20:13:16.2943843Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338)
> 2020-01-10T20:13:16.2943931Z - locked <0x00000000c583a5e8> (a org.apache.hadoop.fs.azurebfs.services.ReadBufferManager)
> 2020-01-10T20:13:16.2944030Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56)
> 2020-01-10T20:13:16.2944110Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2944148Z
> 2020-01-10T20:13:16.2944222Z "ABFS-prefetch-3" #158 daemon prio=5 os_prio=0 tid=0x000000001f6b1000 nid=0x17f4 in Object.wait() [0x000000003a33f000]
> 2020-01-10T20:13:16.2944307Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.2944719Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2944803Z at java.lang.Object.wait(Object.java:502)
> 2020-01-10T20:13:16.2944890Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338)
> 2020-01-10T20:13:16.2945002Z - locked <0x00000000c583a5e8> (a org.apache.hadoop.fs.azurebfs.services.ReadBufferManager)
> 2020-01-10T20:13:16.2945093Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56)
> 2020-01-10T20:13:16.2945182Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2945313Z
> 2020-01-10T20:13:16.2945408Z "ABFS-prefetch-2" #157 daemon prio=5 os_prio=0 tid=0x000000001f6ad800 nid=0x21f4 in Object.wait() [0x0000000039f3e000]
> 2020-01-10T20:13:16.2945494Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.2945571Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2945640Z at java.lang.Object.wait(Object.java:502)
> 2020-01-10T20:13:16.2945845Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338)
> 2020-01-10T20:13:16.2946012Z - locked <0x00000000c583a5e8> (a org.apache.hadoop.fs.azurebfs.services.ReadBufferManager)
> 2020-01-10T20:13:16.2946210Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56)
> 2020-01-10T20:13:16.2946379Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2946515Z
> 2020-01-10T20:13:16.2946738Z "ABFS-prefetch-1" #156 daemon prio=5 os_prio=0 tid=0x000000001f6af000 nid=0x3274 in Object.wait() [0x0000000039b3f000]
> 2020-01-10T20:13:16.2947028Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.2947182Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2947265Z at java.lang.Object.wait(Object.java:502)
> 2020-01-10T20:13:16.2947352Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338)
> 2020-01-10T20:13:16.2947463Z - locked <0x00000000c583a5e8> (a org.apache.hadoop.fs.azurebfs.services.ReadBufferManager)
> 2020-01-10T20:13:16.2947553Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56)
> 2020-01-10T20:13:16.2947634Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2947676Z
> 2020-01-10T20:13:16.2947760Z "ABFS-prefetch-0" #155 daemon prio=5 os_prio=0 tid=0x000000001f6ac800 nid=0x5970 in Object.wait() [0x000000003973e000]
> 2020-01-10T20:13:16.2947845Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.2947921Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2947996Z at java.lang.Object.wait(Object.java:502)
> 2020-01-10T20:13:16.2948087Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338)
> 2020-01-10T20:13:16.2948180Z - locked <0x00000000c583a5e8> (a org.apache.hadoop.fs.azurebfs.services.ReadBufferManager)
> 2020-01-10T20:13:16.2948449Z at org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56)
> 2020-01-10T20:13:16.2948531Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2948577Z
> 2020-01-10T20:13:16.2948662Z "abfs-timer-client-throttling-analyzer-write" #154 daemon prio=5 os_prio=0 tid=0x000000001f6af800 nid=0x2cc0 in Object.wait() [0x000000003933e000]
> 2020-01-10T20:13:16.2948765Z java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 2020-01-10T20:13:16.2948835Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2948914Z at java.util.TimerThread.mainLoop(Timer.java:552)
> 2020-01-10T20:13:16.2948991Z - locked <0x00000000c57ba7f0> (a java.util.TaskQueue)
> 2020-01-10T20:13:16.2949070Z at java.util.TimerThread.run(Timer.java:505)
> 2020-01-10T20:13:16.2949111Z
> 2020-01-10T20:13:16.2949195Z "abfs-timer-client-throttling-analyzer-read" #153 daemon prio=5 os_prio=0 tid=0x000000001f7e4000 nid=0x37d0 in Object.wait() [0x0000000038f3e000]
> 2020-01-10T20:13:16.2949301Z java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 2020-01-10T20:13:16.2949695Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2949765Z at java.util.TimerThread.mainLoop(Timer.java:552)
> 2020-01-10T20:13:16.2949847Z - locked <0x00000000c5c3e478> (a java.util.TaskQueue)
> 2020-01-10T20:13:16.2949918Z at java.util.TimerThread.run(Timer.java:505)
> 2020-01-10T20:13:16.2949960Z
> 2020-01-10T20:13:16.2950064Z "ForkJoinPool-1-worker-9" #151 daemon prio=5 os_prio=0 tid=0x000000001f7e3800 nid=0x6338 waiting on condition [0x000000002d33f000]
> 2020-01-10T20:13:16.2950150Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2950312Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2950396Z - parking to wait for <0x00000000c5757d20> (a scala.concurrent.forkjoin.ForkJoinPool)
> 2020-01-10T20:13:16.2950493Z at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 2020-01-10T20:13:16.2950579Z at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 2020-01-10T20:13:16.2950683Z at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2020-01-10T20:13:16.2950733Z
> 2020-01-10T20:13:16.2950835Z "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #150 daemon prio=5 os_prio=0 tid=0x000000001f7e2000 nid=0x5b48 in Object.wait() [0x000000002953f000]
> 2020-01-10T20:13:16.2950933Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.2951010Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2951086Z at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> 2020-01-10T20:13:16.2951182Z - locked <0x00000000c58260c8> (a java.lang.ref.ReferenceQueue$Lock)
> 2020-01-10T20:13:16.2951263Z at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
> 2020-01-10T20:13:16.2951361Z at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3762)
> 2020-01-10T20:13:16.2951444Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2951502Z
> 2020-01-10T20:13:16.2951587Z "task-result-getter-0" #149 daemon prio=5 os_prio=0 tid=0x000000001f7e1000 nid=0x58e8 waiting on condition [0x000000002913f000]
> 2020-01-10T20:13:16.2951686Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2951755Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2951856Z - parking to wait for <0x00000000c3ce7cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2951948Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2952059Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2952162Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.2952259Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2952348Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2952452Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2952528Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2952577Z
> 2020-01-10T20:13:16.2952657Z "pool-30-thread-16" #147 prio=5 os_prio=0 tid=0x000000001f7e0800 nid=0x3c1c waiting on condition [0x000000002873e000]
> 2020-01-10T20:13:16.2952907Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2952974Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2953069Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2953164Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2953262Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2953361Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2953538Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2953635Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2953721Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2953817Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2953889Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2953991Z
> 2020-01-10T20:13:16.2954084Z "pool-30-thread-15" #146 prio=5 os_prio=0 tid=0x000000001f7de000 nid=0x3578 waiting on condition [0x000000002833e000]
> 2020-01-10T20:13:16.2954336Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2954419Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2954508Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2954614Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2954874Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2955166Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2955265Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2955351Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2955452Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2955704Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2955786Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2955836Z
> 2020-01-10T20:13:16.2955915Z "pool-30-thread-14" #145 prio=5 os_prio=0 tid=0x000000001f7df800 nid=0x6048 waiting on condition [0x0000000027f3e000]
> 2020-01-10T20:13:16.2956007Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2956082Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2956172Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2956270Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2956365Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2956478Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2956577Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2956676Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2956770Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2956863Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2956937Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2956985Z
> 2020-01-10T20:13:16.2957065Z "pool-30-thread-13" #144 prio=5 os_prio=0 tid=0x000000001f7df000 nid=0x6a8 waiting on condition [0x0000000027b3e000]
> 2020-01-10T20:13:16.2957154Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2957229Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2957323Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2957413Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2957515Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2957693Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2957803Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2957896Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2957993Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2958080Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2958232Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2958275Z
> 2020-01-10T20:13:16.2958370Z "pool-30-thread-12" #143 prio=5 os_prio=0 tid=0x000000001f7dd800 nid=0x1990 waiting on condition [0x000000002773e000]
> 2020-01-10T20:13:16.2958452Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2958538Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2958629Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2958736Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2958989Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2959275Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2959375Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2959629Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2959714Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2959792Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2959872Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2959910Z
> 2020-01-10T20:13:16.2959983Z "pool-30-thread-11" #142 prio=5 os_prio=0 tid=0x000000001f7dc800 nid=0x3a8c waiting on condition [0x000000002733f000]
> 2020-01-10T20:13:16.2960061Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2960122Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2960209Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2960296Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2960385Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2960481Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2960576Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2960663Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2960741Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2960829Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2960895Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2960938Z
> 2020-01-10T20:13:16.2961010Z "pool-30-thread-10" #141 prio=5 os_prio=0 tid=0x000000001f7d6000 nid=0x1334 waiting on condition [0x0000000026f3e000]
> 2020-01-10T20:13:16.2961096Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2961158Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2961244Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2961403Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2961494Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2961585Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2961681Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2961764Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2961909Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2961989Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2962063Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2962101Z
> 2020-01-10T20:13:16.2962179Z "pool-30-thread-9" #140 prio=5 os_prio=0 tid=0x000000001f7d9000 nid=0x31cc waiting on condition [0x0000000026b3f000]
> 2020-01-10T20:13:16.2962261Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2962331Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2962412Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2962499Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2962582Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2962688Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2962777Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2962868Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2962953Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2963039Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2963106Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2963151Z
> 2020-01-10T20:13:16.2963223Z "pool-30-thread-8" #139 prio=5 os_prio=0 tid=0x000000001f7d8000 nid=0x43f4 waiting for monitor entry [0x000000002673e000]
> 2020-01-10T20:13:16.2963310Z java.lang.Thread.State: BLOCKED (on object monitor)
> 2020-01-10T20:13:16.2963587Z at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:30)
> 2020-01-10T20:13:16.2963687Z at com.microsoft.azure.eventhubs.impl.MessageSender.onClose(MessageSender.java:424)
> 2020-01-10T20:13:16.2963771Z at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:70)
> 2020-01-10T20:13:16.2963864Z at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.handleRemoteLinkClosed(BaseLinkHandler.java:106)
> 2020-01-10T20:13:16.2963963Z at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:48)
> 2020-01-10T20:13:16.2964045Z at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
> 2020-01-10T20:13:16.2964129Z at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
> 2020-01-10T20:13:16.2964254Z at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
> 2020-01-10T20:13:16.2964335Z at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
> 2020-01-10T20:13:16.2964431Z at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:512)
> 2020-01-10T20:13:16.2964512Z at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2020-01-10T20:13:16.2964592Z at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2020-01-10T20:13:16.2964678Z at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> 2020-01-10T20:13:16.2965031Z at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> 2020-01-10T20:13:16.2965116Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.2965201Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2965483Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2965528Z
> 2020-01-10T20:13:16.2965604Z "pool-30-thread-7" #138 prio=5 os_prio=0 tid=0x000000001f7d5000 nid=0x32d8 waiting on condition [0x000000002633e000]
> 2020-01-10T20:13:16.2965752Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2965818Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2965907Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2965992Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2966094Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2966189Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2966288Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2966379Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2966469Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2966556Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2966626Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2966837Z
> 2020-01-10T20:13:16.2966923Z "pool-30-thread-6" #137 prio=5 os_prio=0 tid=0x000000001f7dc000 nid=0x62a0 waiting on condition [0x0000000025f3e000]
> 2020-01-10T20:13:16.2967012Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2967089Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2967178Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2967278Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2967378Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2967487Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2967598Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2967690Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2967783Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2967955Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2968076Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2968119Z
> 2020-01-10T20:13:16.2968206Z "pool-30-thread-5" #136 prio=5 os_prio=0 tid=0x000000001f7da800 nid=0x3f24 waiting on condition [0x0000000025b3e000]
> 2020-01-10T20:13:16.2968289Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2968363Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2968461Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2968557Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2968651Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2968757Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2968947Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2969049Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2969138Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2969231Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2969373Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2969423Z
> 2020-01-10T20:13:16.2969504Z "pool-30-thread-4" #135 prio=5 os_prio=0 tid=0x000000001f7db000 nid=0x22ec waiting on condition [0x00000000252bf000]
> 2020-01-10T20:13:16.2969592Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2969661Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2969756Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2969854Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2970118Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2970400Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2970496Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2970744Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2970830Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2970907Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2970986Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2971031Z
> 2020-01-10T20:13:16.2971110Z "pool-30-thread-3" #134 prio=5 os_prio=0 tid=0x000000001f7d7800 nid=0x56ec waiting on condition [0x0000000024ebe000]
> 2020-01-10T20:13:16.2971183Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2971251Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2971331Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2971416Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2971506Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2971606Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2971694Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2971797Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2971889Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2971967Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2972049Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2972086Z
> 2020-01-10T20:13:16.2972156Z "pool-30-thread-2" #133 prio=5 os_prio=0 tid=0x000000001f7d6800 nid=0x4d18 waiting on condition [0x0000000024abf000]
> 2020-01-10T20:13:16.2972248Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2972310Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2972405Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2972490Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2972645Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2972743Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2972834Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2972923Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2973002Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2973148Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2973216Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2973259Z
> 2020-01-10T20:13:16.2973330Z "pool-30-thread-1" #132 prio=5 os_prio=0 tid=0x0000000019ee7800 nid=0x6390 waiting on condition [0x000000003873f000]
> 2020-01-10T20:13:16.2973417Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2973480Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2973574Z - parking to wait for <0x00000000c3d74118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2973655Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2973753Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2973843Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-01-10T20:13:16.2973954Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2974036Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2974121Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2974207Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2974281Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2974318Z
> 2020-01-10T20:13:16.2974401Z "spark-listener-group-streams" #130 daemon prio=5 os_prio=0 tid=0x00000000134f5800 nid=0x2ba0 waiting on condition [0x0000000037f3e000]
> 2020-01-10T20:13:16.2974481Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2974549Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2974629Z - parking to wait for <0x00000000c3a32880> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2974722Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2974806Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2974896Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.2974998Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97)
> 2020-01-10T20:13:16.2975112Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2975223Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2975310Z at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> 2020-01-10T20:13:16.2975416Z at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2975511Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83)
> 2020-01-10T20:13:16.2975611Z at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
> 2020-01-10T20:13:16.2975767Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)
> 2020-01-10T20:13:16.2975819Z
> 2020-01-10T20:13:16.2975897Z "spark-listener-group-shared" #129 daemon prio=5 os_prio=0 tid=0x00000000134f4000 nid=0x3010 waiting on condition [0x0000000037b3e000]
> 2020-01-10T20:13:16.2975980Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2976042Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2976130Z - parking to wait for <0x00000000c3d10230> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2976268Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2976359Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2976619Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.2976725Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97)
> 2020-01-10T20:13:16.2976842Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2976952Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2977041Z at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> 2020-01-10T20:13:16.2977136Z at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2977246Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83)
> 2020-01-10T20:13:16.2977331Z at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
> 2020-01-10T20:13:16.2977419Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)
> 2020-01-10T20:13:16.2977473Z
> 2020-01-10T20:13:16.2977552Z "element-tracking-store-worker" #128 daemon prio=5 os_prio=0 tid=0x00000000134fa000 nid=0x434c waiting on condition [0x000000003773f000]
> 2020-01-10T20:13:16.2977640Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2977709Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2977965Z - parking to wait for <0x00000000c3b39828> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2978062Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2978167Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2978266Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.2978404Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2978511Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2978598Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2978678Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2978720Z
> 2020-01-10T20:13:16.2978808Z "spark-listener-group-executorManagement" #118 daemon prio=5 os_prio=0 tid=0x00000000134fb000 nid=0xaf0 waiting on condition [0x000000003733f000]
> 2020-01-10T20:13:16.2978904Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2978977Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2979075Z - parking to wait for <0x00000000c3b3a818> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2979171Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2979265Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2979459Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.2979564Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97)
> 2020-01-10T20:13:16.2979689Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2979803Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2979973Z at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> 2020-01-10T20:13:16.2980071Z at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2980180Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83)
> 2020-01-10T20:13:16.2980285Z at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
> 2020-01-10T20:13:16.2980373Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)
> 2020-01-10T20:13:16.2980423Z
> 2020-01-10T20:13:16.2980516Z "spark-listener-group-appStatus" #90 daemon prio=5 os_prio=0 tid=0x00000000134f2800 nid=0x1f30 waiting on condition [0x0000000036f3e000]
> 2020-01-10T20:13:16.2980603Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2980679Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2980776Z - parking to wait for <0x00000000c3a74eb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2980873Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2980974Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2981228Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.2981345Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97)
> 2020-01-10T20:13:16.2981456Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2981725Z at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2981820Z at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> 2020-01-10T20:13:16.2981923Z at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)
> 2020-01-10T20:13:16.2982178Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83)
> 2020-01-10T20:13:16.2982272Z at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
> 2020-01-10T20:13:16.2982351Z at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)
> 2020-01-10T20:13:16.2982402Z
> 2020-01-10T20:13:16.2982479Z "context-cleaner-periodic-gc" #127 daemon prio=5 os_prio=0 tid=0x00000000134f9800 nid=0x51c8 waiting on condition [0x0000000036b3f000]
> 2020-01-10T20:13:16.2982562Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-01-10T20:13:16.2982626Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2982711Z - parking to wait for <0x00000000c3d74480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2982800Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-01-10T20:13:16.2982893Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-01-10T20:13:16.2982987Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-01-10T20:13:16.2983153Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2983244Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2983324Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2983407Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2983475Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2983573Z
> 2020-01-10T20:13:16.2983657Z "Spark Context Cleaner" #126 daemon prio=5 os_prio=0 tid=0x00000000134f8000 nid=0x620c waiting for monitor entry [0x000000003673c000]
> 2020-01-10T20:13:16.2983736Z java.lang.Thread.State: BLOCKED (on object monitor)
> 2020-01-10T20:13:16.2983814Z at java.lang.ClassLoader.checkCerts(ClassLoader.java:887)
> 2020-01-10T20:13:16.2983893Z - waiting to lock <0x00000000ca33e4c8> (a sbt.internal.ManagedClassLoader$ZombieClassLoader)
> 2020-01-10T20:13:16.2983986Z at java.lang.ClassLoader.preDefineClass(ClassLoader.java:668)
> 2020-01-10T20:13:16.2984063Z at java.lang.ClassLoader.defineClass(ClassLoader.java:761)
> 2020-01-10T20:13:16.2984301Z at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> 2020-01-10T20:13:16.2984612Z at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> 2020-01-10T20:13:16.2984682Z at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> 2020-01-10T20:13:16.2984755Z at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> 2020-01-10T20:13:16.2984825Z at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> 2020-01-10T20:13:16.2984897Z at java.security.AccessController.doPrivileged(Native Method)
> 2020-01-10T20:13:16.2984964Z at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> 2020-01-10T20:13:16.2985042Z at sbt.internal.ManagedClassLoader$ZombieClassLoader.lookupClass(LayeredClassLoaders.scala:336)
> 2020-01-10T20:13:16.2985301Z at sbt.internal.ManagedClassLoader.findClass(LayeredClassLoaders.scala:375)
> 2020-01-10T20:13:16.2985617Z at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 2020-01-10T20:13:16.2985689Z - locked <0x00000000c1f359f0> (a sbt.internal.LayeredClassLoader)
> 2020-01-10T20:13:16.2985767Z at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 2020-01-10T20:13:16.2985845Z at org.apache.spark.storage.BlockManagerMaster.removeShuffle(BlockManagerMaster.scala:138)
> 2020-01-10T20:13:16.2985932Z at org.apache.spark.ContextCleaner.doCleanupShuffle(ContextCleaner.scala:226)
> 2020-01-10T20:13:16.2986033Z at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:192)
> 2020-01-10T20:13:16.2986143Z at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:185)
> 2020-01-10T20:13:16.2986228Z at scala.Option.foreach(Option.scala:257)
> 2020-01-10T20:13:16.2986320Z at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:185)
> 2020-01-10T20:13:16.2986404Z - locked <0x00000000c3d74cd0> (a org.apache.spark.ContextCleaner)
> 2020-01-10T20:13:16.2986486Z at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
> 2020-01-10T20:13:16.2986573Z at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:178)
> 2020-01-10T20:13:16.2986655Z at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73)
> 2020-01-10T20:13:16.2986704Z
> 2020-01-10T20:13:16.2986783Z "shuffle-server-5-1" #125 daemon prio=5 os_prio=0 tid=0x00000000134f8800 nid=0x1e2c runnable [0x000000003633e000]
> 2020-01-10T20:13:16.2986857Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.2986931Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> 2020-01-10T20:13:16.2987007Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> 2020-01-10T20:13:16.2987162Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> 2020-01-10T20:13:16.2987241Z at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> 2020-01-10T20:13:16.2987322Z at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 2020-01-10T20:13:16.2987397Z - locked <0x00000000c3c203a0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 2020-01-10T20:13:16.2987483Z - locked <0x00000000c3c551c0> (a java.util.Collections$UnmodifiableSet)
> 2020-01-10T20:13:16.2987804Z - locked <0x00000000c3c202d0> (a sun.nio.ch.WindowsSelectorImpl)
> 2020-01-10T20:13:16.2987877Z at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 2020-01-10T20:13:16.2987967Z at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
> 2020-01-10T20:13:16.2988051Z at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
> 2020-01-10T20:13:16.2988137Z at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
> 2020-01-10T20:13:16.2988220Z at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
> 2020-01-10T20:13:16.2988317Z at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
> 2020-01-10T20:13:16.2988393Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2988437Z
> 2020-01-10T20:13:16.2988514Z "driver-heartbeater" #124 daemon prio=5 os_prio=0 tid=0x00000000134f2000 nid=0x3940 waiting on condition [0x0000000035f3f000]
> 2020-01-10T20:13:16.2988606Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-01-10T20:13:16.2988671Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2988760Z - parking to wait for <0x00000000c3d00fc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2988847Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-01-10T20:13:16.2989138Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-01-10T20:13:16.2989242Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-01-10T20:13:16.2989351Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2989444Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2989540Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2989636Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2989717Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2989760Z
> 2020-01-10T20:13:16.2989849Z "dag-scheduler-event-loop" #123 daemon prio=5 os_prio=0 tid=0x00000000134f0800 nid=0x3a40 waiting on condition [0x0000000035b3f000]
> 2020-01-10T20:13:16.2989944Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.2990020Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2990108Z - parking to wait for <0x00000000c3ce7280> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2990208Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.2990303Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.2990405Z at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> 2020-01-10T20:13:16.2990499Z at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> 2020-01-10T20:13:16.2990590Z at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:47)
> 2020-01-10T20:13:16.2990637Z
> 2020-01-10T20:13:16.2990727Z "netty-rpc-env-timeout" #122 daemon prio=5 os_prio=0 tid=0x00000000134f1000 nid=0x296c waiting on condition [0x000000003573e000]
> 2020-01-10T20:13:16.2990889Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-01-10T20:13:16.2990966Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2991057Z - parking to wait for <0x00000000c3dbd078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2991157Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-01-10T20:13:16.2991255Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-01-10T20:13:16.2991429Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-01-10T20:13:16.2991529Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2991629Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2991724Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2991819Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2991898Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2991942Z
> 2020-01-10T20:13:16.2992021Z "Timer-3" #121 daemon prio=5 os_prio=0 tid=0x00000000134f6800 nid=0xef8 in Object.wait() [0x000000003533e000]
> 2020-01-10T20:13:16.2992111Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.2992182Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2992429Z - waiting on <0x00000000c3c4b628> (a java.util.TaskQueue)
> 2020-01-10T20:13:16.2992667Z at java.lang.Object.wait(Object.java:502)
> 2020-01-10T20:13:16.2992741Z at java.util.TimerThread.mainLoop(Timer.java:526)
> 2020-01-10T20:13:16.2992809Z - locked <0x00000000c3c4b628> (a java.util.TaskQueue)
> 2020-01-10T20:13:16.2993043Z at java.util.TimerThread.run(Timer.java:505)
> 2020-01-10T20:13:16.2993088Z
> 2020-01-10T20:13:16.2993162Z "Timer-2" #120 daemon prio=5 os_prio=0 tid=0x00000000134f3800 nid=0x163c in Object.wait() [0x0000000034f3f000]
> 2020-01-10T20:13:16.2993238Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.2993304Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.2993370Z - waiting on <0x00000000c3c4aca0> (a java.util.TaskQueue)
> 2020-01-10T20:13:16.2993436Z at java.lang.Object.wait(Object.java:502)
> 2020-01-10T20:13:16.2993498Z at java.util.TimerThread.mainLoop(Timer.java:526)
> 2020-01-10T20:13:16.2993569Z - locked <0x00000000c3c4aca0> (a java.util.TaskQueue)
> 2020-01-10T20:13:16.2993638Z at java.util.TimerThread.run(Timer.java:505)
> 2020-01-10T20:13:16.2993676Z
> 2020-01-10T20:13:16.2993759Z "heartbeat-receiver-event-loop-thread" #119 daemon prio=5 os_prio=0 tid=0x00000000134f7000 nid=0x4984 waiting on condition [0x0000000034b3f000]
> 2020-01-10T20:13:16.2993844Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-01-10T20:13:16.2993913Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2993999Z - parking to wait for <0x00000000c3de5808> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2994245Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-01-10T20:13:16.2994335Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-01-10T20:13:16.2994425Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-01-10T20:13:16.2994523Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-01-10T20:13:16.2994604Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-01-10T20:13:16.2994686Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-01-10T20:13:16.2994761Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.2994902Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2994940Z
> 2020-01-10T20:13:16.2995017Z "SparkUI-117" #117 daemon prio=5 os_prio=0 tid=0x0000000012be9800 nid=0x6208 waiting on condition [0x000000003473f000]
> 2020-01-10T20:13:16.2995090Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-01-10T20:13:16.2995155Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2995234Z - parking to wait for <0x00000000c3de5f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2995376Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-01-10T20:13:16.2995461Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-01-10T20:13:16.2995550Z at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
> 2020-01-10T20:13:16.2995638Z at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
> 2020-01-10T20:13:16.2995727Z at org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
> 2020-01-10T20:13:16.2995808Z at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
> 2020-01-10T20:13:16.2995882Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2995920Z
> 2020-01-10T20:13:16.2995994Z "SparkUI-116" #116 daemon prio=5 os_prio=0 tid=0x0000000012be2000 nid=0x3b34 waiting on condition [0x000000003433e000]
> 2020-01-10T20:13:16.2996072Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-01-10T20:13:16.2996139Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2996218Z - parking to wait for <0x00000000c3de5f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2996304Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-01-10T20:13:16.2996394Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-01-10T20:13:16.2996483Z at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
> 2020-01-10T20:13:16.2996564Z at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
> 2020-01-10T20:13:16.2996650Z at org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
> 2020-01-10T20:13:16.2996730Z at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
> 2020-01-10T20:13:16.2996810Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2996846Z
> 2020-01-10T20:13:16.2996935Z "SparkUI-115-acceptor-0@4be1e6e6-ServerConnector@452601ea\{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}" #115 daemon prio=3 os_prio=-1 tid=0x0000000012be9000 nid=0x3a54 runnable [0x0000000033f3e000]
> 2020-01-10T20:13:16.2997016Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.2997094Z at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> 2020-01-10T20:13:16.2997168Z at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> 2020-01-10T20:13:16.2997250Z at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> 2020-01-10T20:13:16.2997317Z - locked <0x00000000c3e3e9a8> (a java.lang.Object)
> 2020-01-10T20:13:16.2997397Z at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:397)
> 2020-01-10T20:13:16.2997477Z at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
> 2020-01-10T20:13:16.2997571Z at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> 2020-01-10T20:13:16.2997656Z at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> 2020-01-10T20:13:16.2997724Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2997760Z
> 2020-01-10T20:13:16.2997836Z "SparkUI-114" #114 daemon prio=5 os_prio=0 tid=0x0000000012be6000 nid=0x5474 waiting on condition [0x0000000033b3f000]
> 2020-01-10T20:13:16.2997974Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-01-10T20:13:16.2998040Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.2998118Z - parking to wait for <0x00000000c3de5f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.2998204Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-01-10T20:13:16.2998288Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-01-10T20:13:16.2998439Z at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
> 2020-01-10T20:13:16.2998527Z at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
> 2020-01-10T20:13:16.2998609Z at org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
> 2020-01-10T20:13:16.2998702Z at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
> 2020-01-10T20:13:16.2998768Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.2998806Z
> 2020-01-10T20:13:16.2998878Z "SparkUI-113" #113 daemon prio=5 os_prio=0 tid=0x0000000012be3800 nid=0x22b4 runnable [0x000000003373e000]
> 2020-01-10T20:13:16.2998946Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.2999024Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> 2020-01-10T20:13:16.2999098Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> 2020-01-10T20:13:16.2999189Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> 2020-01-10T20:13:16.2999270Z at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> 2020-01-10T20:13:16.2999341Z at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 2020-01-10T20:13:16.2999411Z - locked <0x00000000c3e3aac0> (a sun.nio.ch.Util$3)
> 2020-01-10T20:13:16.2999488Z - locked <0x00000000c3e3aab0> (a java.util.Collections$UnmodifiableSet)
> 2020-01-10T20:13:16.2999564Z - locked <0x00000000c3e3a960> (a sun.nio.ch.WindowsSelectorImpl)
> 2020-01-10T20:13:16.2999631Z at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 2020-01-10T20:13:16.2999702Z at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> 2020-01-10T20:13:16.2999779Z at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
> 2020-01-10T20:13:16.2999871Z at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
> 2020-01-10T20:13:16.2999966Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
> 2020-01-10T20:13:16.3000064Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> 2020-01-10T20:13:16.3000154Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
> 2020-01-10T20:13:16.3000245Z at org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
> 2020-01-10T20:13:16.3000326Z at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> 2020-01-10T20:13:16.3000410Z at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> 2020-01-10T20:13:16.3000477Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3000520Z
> 2020-01-10T20:13:16.3000588Z "SparkUI-112" #112 daemon prio=5 os_prio=0 tid=0x0000000012be6800 nid=0x21c4 runnable [0x000000003333e000]
> 2020-01-10T20:13:16.3000667Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.3000732Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> 2020-01-10T20:13:16.3000813Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> 2020-01-10T20:13:16.3000891Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> 2020-01-10T20:13:16.3001039Z at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> 2020-01-10T20:13:16.3001111Z at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 2020-01-10T20:13:16.3001182Z - locked <0x00000000c3de66a8> (a sun.nio.ch.Util$3)
> 2020-01-10T20:13:16.3001252Z - locked <0x00000000c3de6698> (a java.util.Collections$UnmodifiableSet)
> 2020-01-10T20:13:16.3001328Z - locked <0x00000000c3de6548> (a sun.nio.ch.WindowsSelectorImpl)
> 2020-01-10T20:13:16.3001396Z at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 2020-01-10T20:13:16.3001467Z at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> 2020-01-10T20:13:16.3001604Z at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
> 2020-01-10T20:13:16.3001693Z at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
> 2020-01-10T20:13:16.3001940Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
> 2020-01-10T20:13:16.3002040Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> 2020-01-10T20:13:16.3002129Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
> 2020-01-10T20:13:16.3002206Z at org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
> 2020-01-10T20:13:16.3002289Z at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> 2020-01-10T20:13:16.3002367Z at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> 2020-01-10T20:13:16.3002442Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3002478Z
> 2020-01-10T20:13:16.3002543Z "SparkUI-111" #111 daemon prio=5 os_prio=0 tid=0x0000000012be3000 nid=0x1cbc runnable [0x0000000032f3e000]
> 2020-01-10T20:13:16.3002615Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.3002686Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> 2020-01-10T20:13:16.3002764Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> 2020-01-10T20:13:16.3002852Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> 2020-01-10T20:13:16.3002924Z at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> 2020-01-10T20:13:16.3002998Z at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 2020-01-10T20:13:16.3003062Z - locked <0x00000000c3dea598> (a sun.nio.ch.Util$3)
> 2020-01-10T20:13:16.3003134Z - locked <0x00000000c3dea588> (a java.util.Collections$UnmodifiableSet)
> 2020-01-10T20:13:16.3003208Z - locked <0x00000000c3dea438> (a sun.nio.ch.WindowsSelectorImpl)
> 2020-01-10T20:13:16.3003278Z at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 2020-01-10T20:13:16.3003342Z at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> 2020-01-10T20:13:16.3003420Z at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
> 2020-01-10T20:13:16.3003506Z at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
> 2020-01-10T20:13:16.3003596Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
> 2020-01-10T20:13:16.3003683Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> 2020-01-10T20:13:16.3003774Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
> 2020-01-10T20:13:16.3003856Z at org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
> 2020-01-10T20:13:16.3003939Z at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> 2020-01-10T20:13:16.3004016Z at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> 2020-01-10T20:13:16.3004085Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3004182Z
> 2020-01-10T20:13:16.3004254Z "SparkUI-110" #110 daemon prio=5 os_prio=0 tid=0x0000000012be4800 nid=0x3920 runnable [0x0000000032b3e000]
> 2020-01-10T20:13:16.3004321Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.3004390Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> 2020-01-10T20:13:16.3004462Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> 2020-01-10T20:13:16.3004543Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> 2020-01-10T20:13:16.3004616Z at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> 2020-01-10T20:13:16.3004749Z at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 2020-01-10T20:13:16.3004814Z - locked <0x00000000c3e3f0c8> (a sun.nio.ch.Util$3)
> 2020-01-10T20:13:16.3004887Z - locked <0x00000000c3e3f0b8> (a java.util.Collections$UnmodifiableSet)
> 2020-01-10T20:13:16.3004956Z - locked <0x00000000c3e3ef68> (a sun.nio.ch.WindowsSelectorImpl)
> 2020-01-10T20:13:16.3005802Z at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 2020-01-10T20:13:16.3005881Z at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
> 2020-01-10T20:13:16.3005976Z at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
> 2020-01-10T20:13:16.3006084Z at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
> 2020-01-10T20:13:16.3006188Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
> 2020-01-10T20:13:16.3006307Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> 2020-01-10T20:13:16.3006409Z at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
> 2020-01-10T20:13:16.3006507Z at org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
> 2020-01-10T20:13:16.3006605Z at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> 2020-01-10T20:13:16.3006703Z at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> 2020-01-10T20:13:16.3006780Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3006829Z
> 2020-01-10T20:13:16.3006915Z "RemoteBlock-temp-file-clean-thread" #109 daemon prio=5 os_prio=0 tid=0x0000000012be5000 nid=0x3978 in Object.wait() [0x000000003273f000]
> 2020-01-10T20:13:16.3007014Z java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 2020-01-10T20:13:16.3007093Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.3007175Z at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> 2020-01-10T20:13:16.3007257Z - locked <0x00000000c3cfec80> (a java.lang.ref.ReferenceQueue$Lock)
> 2020-01-10T20:13:16.3007382Z at org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager.org$apache$spark$storage$BlockManager$RemoteBlockDownloadFileManager$$keepCleaning(BlockManager.scala:1724)
> 2020-01-10T20:13:16.3007504Z at org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager$$anon$2.run(BlockManager.scala:1692)
> 2020-01-10T20:13:16.3007564Z
> 2020-01-10T20:13:16.3007649Z "map-output-dispatcher-7" #107 daemon prio=5 os_prio=0 tid=0x0000000012be8000 nid=0x1dc0 waiting on condition [0x000000003233e000]
> 2020-01-10T20:13:16.3007740Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3007810Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3007905Z - parking to wait for <0x00000000c3b84b08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3008004Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3008105Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3008198Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3008415Z at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
> 2020-01-10T20:13:16.3008508Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3008602Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3008678Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3008867Z
> 2020-01-10T20:13:16.3009124Z "map-output-dispatcher-6" #106 daemon prio=5 os_prio=0 tid=0x0000000012be7800 nid=0x4930 waiting on condition [0x0000000031f3e000]
> 2020-01-10T20:13:16.3010302Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3010365Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3010448Z - parking to wait for <0x00000000c3b84b08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3010525Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3010620Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3010700Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3010783Z at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
> 2020-01-10T20:13:16.3010858Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3010938Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3011015Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3011052Z
> 2020-01-10T20:13:16.3011122Z "map-output-dispatcher-5" #105 daemon prio=5 os_prio=0 tid=0x0000000012bdb800 nid=0x320c waiting on condition [0x0000000031b3f000]
> 2020-01-10T20:13:16.3011200Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3011259Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3011347Z - parking to wait for <0x00000000c3b84b08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3011423Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3011683Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3011765Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3011848Z at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
> 2020-01-10T20:13:16.3011937Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3012013Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3012083Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3012120Z
> 2020-01-10T20:13:16.3012192Z "map-output-dispatcher-4" #104 daemon prio=5 os_prio=0 tid=0x0000000012bda800 nid=0x1180 waiting on condition [0x000000003173f000]
> 2020-01-10T20:13:16.3012282Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3012342Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3012427Z - parking to wait for <0x00000000c3b84b08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3012505Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3012592Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3012684Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3012761Z at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
> 2020-01-10T20:13:16.3012842Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3012919Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3013075Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3013111Z
> 2020-01-10T20:13:16.3013183Z "map-output-dispatcher-3" #103 daemon prio=5 os_prio=0 tid=0x0000000012bdc000 nid=0x27f4 waiting on condition [0x000000003133f000]
> 2020-01-10T20:13:16.3013265Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3013325Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3013409Z - parking to wait for <0x00000000c3b84b08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3014283Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3014372Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3014624Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3014699Z at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
> 2020-01-10T20:13:16.3014792Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3014866Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3014941Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3014976Z
> 2020-01-10T20:13:16.3015046Z "map-output-dispatcher-2" #102 daemon prio=5 os_prio=0 tid=0x0000000012bdd000 nid=0x2e04 waiting on condition [0x0000000030f3e000]
> 2020-01-10T20:13:16.3015127Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3015197Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3015273Z - parking to wait for <0x00000000c3b84b08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3015356Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3015435Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3015525Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3015599Z at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
> 2020-01-10T20:13:16.3015681Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3015754Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3015829Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3015870Z
> 2020-01-10T20:13:16.3015951Z "map-output-dispatcher-1" #101 daemon prio=5 os_prio=0 tid=0x0000000012be1800 nid=0x358c waiting on condition [0x0000000030b3f000]
> 2020-01-10T20:13:16.3016022Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3016091Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3016166Z - parking to wait for <0x00000000c3b84b08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3016263Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3016340Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3016427Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3016500Z at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
> 2020-01-10T20:13:16.3016589Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3016667Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3016741Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3016775Z
> 2020-01-10T20:13:16.3016851Z "map-output-dispatcher-0" #100 daemon prio=5 os_prio=0 tid=0x0000000012bde800 nid=0xfe8 waiting on condition [0x000000003073f000]
> 2020-01-10T20:13:16.3016994Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3017061Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3017136Z - parking to wait for <0x00000000c3b84b08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3017220Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3017297Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3017381Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3017884Z at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
> 2020-01-10T20:13:16.3017970Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3018043Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3018120Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3018156Z
> 2020-01-10T20:13:16.3018229Z "rpc-server-3-1" #99 daemon prio=5 os_prio=0 tid=0x0000000012be0800 nid=0x63d0 runnable [0x000000003033e000]
> 2020-01-10T20:13:16.3018295Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.3018365Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> 2020-01-10T20:13:16.3018437Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> 2020-01-10T20:13:16.3018517Z at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> 2020-01-10T20:13:16.3018598Z at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> 2020-01-10T20:13:16.3018674Z at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 2020-01-10T20:13:16.3018744Z - locked <0x00000000c3e64038> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 2020-01-10T20:13:16.3018821Z - locked <0x00000000c3ec1920> (a java.util.Collections$UnmodifiableSet)
> 2020-01-10T20:13:16.3018897Z - locked <0x00000000c3e63f68> (a sun.nio.ch.WindowsSelectorImpl)
> 2020-01-10T20:13:16.3018968Z at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 2020-01-10T20:13:16.3019043Z at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
> 2020-01-10T20:13:16.3019126Z at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
> 2020-01-10T20:13:16.3019194Z at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
> 2020-01-10T20:13:16.3019278Z at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
> 2020-01-10T20:13:16.3019367Z at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
> 2020-01-10T20:13:16.3019442Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3019477Z
> 2020-01-10T20:13:16.3019553Z "dispatcher-event-loop-7" #98 daemon prio=5 os_prio=0 tid=0x0000000012bdf000 nid=0x58f4 waiting on condition [0x000000002ff3e000]
> 2020-01-10T20:13:16.3019630Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3019700Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3019776Z - parking to wait for <0x00000000c38337c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3019857Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3019937Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3020019Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3020100Z at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
> 2020-01-10T20:13:16.3020182Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3020262Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3020397Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3020433Z
> 2020-01-10T20:13:16.3020510Z "dispatcher-event-loop-6" #97 daemon prio=5 os_prio=0 tid=0x0000000012be0000 nid=0x1b6c waiting on condition [0x000000002fb3f000]
> 2020-01-10T20:13:16.3020580Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3020644Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3020719Z - parking to wait for <0x00000000c38337c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3020800Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3021399Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3021493Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3021573Z at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
> 2020-01-10T20:13:16.3021656Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3021736Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3021799Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3021834Z
> 2020-01-10T20:13:16.3021913Z "dispatcher-event-loop-5" #96 daemon prio=5 os_prio=0 tid=0x0000000012bdd800 nid=0x4888 waiting on condition [0x000000002f73e000]
> 2020-01-10T20:13:16.3021984Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3022050Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3022307Z - parking to wait for <0x00000000c38337c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3022393Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3022479Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3022566Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3022873Z at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
> 2020-01-10T20:13:16.3022954Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3023036Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3023104Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3023141Z
> 2020-01-10T20:13:16.3023229Z "dispatcher-event-loop-4" #95 daemon prio=5 os_prio=0 tid=0x0000000019eea800 nid=0x2b3c waiting on condition [0x000000002f33f000]
> 2020-01-10T20:13:16.3023311Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3023470Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3023550Z - parking to wait for <0x00000000c38337c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3023637Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3023888Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3023970Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3024050Z at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
> 2020-01-10T20:13:16.3024125Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3024207Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3024278Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3024319Z
> 2020-01-10T20:13:16.3024390Z "dispatcher-event-loop-3" #94 daemon prio=5 os_prio=0 tid=0x00000000151b8000 nid=0x6254 waiting on condition [0x000000002ef3f000]
> 2020-01-10T20:13:16.3024467Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3025337Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3025430Z - parking to wait for <0x00000000c38337c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3025678Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3025768Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3025847Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3025928Z at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
> 2020-01-10T20:13:16.3026072Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3026154Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3026216Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3026257Z
> 2020-01-10T20:13:16.3026333Z "dispatcher-event-loop-2" #93 daemon prio=5 os_prio=0 tid=0x00000000151b7000 nid=0x10c4 waiting on condition [0x000000002eb3e000]
> 2020-01-10T20:13:16.3026411Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3026468Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3026549Z - parking to wait for <0x00000000c38337c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3026625Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3026708Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3026793Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3026905Z at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
> 2020-01-10T20:13:16.3026979Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3027063Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3027126Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3027168Z
> 2020-01-10T20:13:16.3027237Z "dispatcher-event-loop-1" #92 daemon prio=5 os_prio=0 tid=0x00000000151b6800 nid=0x5cf4 waiting on condition [0x000000002e73f000]
> 2020-01-10T20:13:16.3027312Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3027370Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3027458Z - parking to wait for <0x00000000c38337c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3027540Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3027625Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3027704Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3027789Z at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
> 2020-01-10T20:13:16.3027862Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3027942Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3028005Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3028046Z
> 2020-01-10T20:13:16.3028115Z "dispatcher-event-loop-0" #91 daemon prio=5 os_prio=0 tid=0x00000000151b5000 nid=0x960 waiting on condition [0x000000002e33e000]
> 2020-01-10T20:13:16.3028197Z java.lang.Thread.State: WAITING (parking)
> 2020-01-10T20:13:16.3028255Z at sun.misc.Unsafe.park(Native Method)
> 2020-01-10T20:13:16.3028336Z - parking to wait for <0x00000000c38337c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-01-10T20:13:16.3028412Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-01-10T20:13:16.3028498Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-01-10T20:13:16.3029049Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-01-10T20:13:16.3029130Z at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
> 2020-01-10T20:13:16.3029203Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3029282Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3029345Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3029452Z
> 2020-01-10T20:13:16.3029524Z "classloader-cache-cleanup-0" #14 daemon prio=5 os_prio=0 tid=0x0000000013e0c000 nid=0x4e24 in Object.wait() [0x00000000196ae000]
> 2020-01-10T20:13:16.3029609Z java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 2020-01-10T20:13:16.3029670Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.3029740Z at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> 2020-01-10T20:13:16.3029816Z - locked <0x00000000c13d6698> (a java.lang.ref.ReferenceQueue$Lock)
> 2020-01-10T20:13:16.3029898Z at sbt.internal.classpath.ClassLoaderCache$CleanupThread.run(ClassLoaderCache.scala:102)
> 2020-01-10T20:13:16.3029942Z
> 2020-01-10T20:13:16.3030012Z "Service Thread" #10 daemon prio=9 os_prio=0 tid=0x000000000f400000 nid=0x346c runnable [0x0000000000000000]
> 2020-01-10T20:13:16.3030080Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.3030114Z
> 2020-01-10T20:13:16.3030189Z "C1 CompilerThread3" #9 daemon prio=9 os_prio=2 tid=0x000000000f36e000 nid=0x37b8 waiting on condition [0x0000000000000000]
> 2020-01-10T20:13:16.3030264Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.3030306Z
> 2020-01-10T20:13:16.3030374Z "C2 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x000000000f36a000 nid=0x5898 waiting on condition [0x0000000000000000]
> 2020-01-10T20:13:16.3030455Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.3030495Z
> 2020-01-10T20:13:16.3030568Z "C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x000000000f368000 nid=0x62dc waiting on condition [0x0000000000000000]
> 2020-01-10T20:13:16.3030636Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.3030669Z
> 2020-01-10T20:13:16.3030743Z "C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x000000000f366800 nid=0x28fc waiting on condition [0x0000000000000000]
> 2020-01-10T20:13:16.3030810Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.3030851Z
> 2020-01-10T20:13:16.3030917Z "Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x000000000f361000 nid=0x4c30 runnable [0x0000000000000000]
> 2020-01-10T20:13:16.3030996Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.3031030Z
> 2020-01-10T20:13:16.3031098Z "Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x000000000f360000 nid=0x3408 waiting on condition [0x0000000000000000]
> 2020-01-10T20:13:16.3031171Z java.lang.Thread.State: RUNNABLE
> 2020-01-10T20:13:16.3031204Z
> 2020-01-10T20:13:16.3031281Z "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x000000000d543800 nid=0x34b4 in Object.wait() [0x000000000ff2f000]
> 2020-01-10T20:13:16.3031352Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.3031417Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.3031483Z at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> 2020-01-10T20:13:16.3031558Z - locked <0x00000000c000bd38> (a java.lang.ref.ReferenceQueue$Lock)
> 2020-01-10T20:13:16.3031627Z at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
> 2020-01-10T20:13:16.3031700Z at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
> 2020-01-10T20:13:16.3031745Z
> 2020-01-10T20:13:16.3031819Z "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x000000000d53c800 nid=0x4f44 in Object.wait() [0x000000000fb2f000]
> 2020-01-10T20:13:16.3031891Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.3031957Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.3032447Z at java.lang.Object.wait(Object.java:502)
> 2020-01-10T20:13:16.3032522Z at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> 2020-01-10T20:13:16.3032589Z - locked <0x00000000c0012c40> (a java.lang.ref.Reference$Lock)
> 2020-01-10T20:13:16.3032665Z at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> 2020-01-10T20:13:16.3032705Z
> 2020-01-10T20:13:16.3032776Z "main" #1 prio=5 os_prio=0 tid=0x00000000015e4800 nid=0x4938 in Object.wait() [0x0000000001bde000]
> 2020-01-10T20:13:16.3032846Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-01-10T20:13:16.3032918Z at java.lang.Object.wait(Native Method)
> 2020-01-10T20:13:16.3033046Z at java.lang.Thread.join(Thread.java:1252)
> 2020-01-10T20:13:16.3033293Z - locked <0x00000000c370a108> (a org.apache.hadoop.util.ShutdownHookManager$1)
> 2020-01-10T20:13:16.3033360Z at java.lang.Thread.join(Thread.java:1326)
> 2020-01-10T20:13:16.3033436Z at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
> 2020-01-10T20:13:16.3033519Z at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
> 2020-01-10T20:13:16.3033592Z at java.lang.Shutdown.runHooks(Shutdown.java:123)
> 2020-01-10T20:13:16.3033653Z at java.lang.Shutdown.sequence(Shutdown.java:167)
> 2020-01-10T20:13:16.3033719Z at java.lang.Shutdown.exit(Shutdown.java:212)
> 2020-01-10T20:13:16.3033787Z - locked <0x00000000c0155de0> (a java.lang.Class for java.lang.Shutdown)
> 2020-01-10T20:13:16.3033860Z at java.lang.Runtime.exit(Runtime.java:109)
> 2020-01-10T20:13:16.3033919Z at java.lang.System.exit(System.java:971)
> 2020-01-10T20:13:16.3033999Z at xsbt.boot.Boot$.xsbt$boot$Boot$$exit(Boot.scala:74)
> 2020-01-10T20:13:16.3034066Z at xsbt.boot.Boot$$anonfun$runImpl$1.apply(Boot.scala:56)
> 2020-01-10T20:13:16.3034139Z at scala.Option.map(Option.scala:145)
> 2020-01-10T20:13:16.3034197Z at xsbt.boot.Boot$.runImpl(Boot.scala:56)
> 2020-01-10T20:13:16.3034267Z at xsbt.boot.Boot$.main(Boot.scala:18)
> 2020-01-10T20:13:16.3034325Z at xsbt.boot.Boot.main(Boot.scala)
> 2020-01-10T20:13:16.3034366Z
> 2020-01-10T20:13:16.3034439Z "VM Thread" os_prio=2 tid=0x000000000d539000 nid=0xd8c runnable
> 2020-01-10T20:13:16.3034478Z
> 2020-01-10T20:13:16.3034552Z "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00000000015fa000 nid=0xf68 runnable
> 2020-01-10T20:13:16.3034597Z
> 2020-01-10T20:13:16.3034675Z "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00000000015fc000 nid=0x4914 runnable
> 2020-01-10T20:13:16.3034719Z
> 2020-01-10T20:13:16.3034780Z "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00000000015fd800 nid=0x4e3c runnable
> 2020-01-10T20:13:16.3034843Z
> 2020-01-10T20:13:16.3034905Z "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000001600000 nid=0x2f54 runnable
> 2020-01-10T20:13:16.3034946Z
> 2020-01-10T20:13:16.3035020Z "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x0000000001601000 nid=0x30b4 runnable
> 2020-01-10T20:13:16.3035062Z
> 2020-01-10T20:13:16.3035135Z "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x0000000001602000 nid=0x4d78 runnable
> 2020-01-10T20:13:16.3035183Z
> 2020-01-10T20:13:16.3035252Z "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x0000000001605800 nid=0x4330 runnable
> 2020-01-10T20:13:16.3035294Z
> 2020-01-10T20:13:16.3035356Z "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x0000000001606800 nid=0x4c94 runnable
> 2020-01-10T20:13:16.3035404Z
> 2020-01-10T20:13:16.3035468Z "VM Periodic Task Thread" os_prio=2 tid=0x000000000f426000 nid=0x3034 waiting on condition
> 2020-01-10T20:13:16.3035513Z
> 2020-01-10T20:13:16.3035567Z JNI global references: 5610
> 2020-01-10T20:13:16.3035601Z
> 2020-01-10T20:13:16.3035628Z
> 2020-01-10T20:13:16.3035689Z Found one Java-level deadlock:
> 2020-01-10T20:13:16.3035745Z =============================
> 2020-01-10T20:13:16.3035806Z "Thread-3":
> 2020-01-10T20:13:16.3035876Z waiting to lock monitor 0x000000001fb7e568 (object 0x00000000c1f359f0, a sbt.internal.LayeredClassLoader),
> 2020-01-10T20:13:16.3035952Z which is held by "Spark Context Cleaner"
> 2020-01-10T20:13:16.3036009Z "Spark Context Cleaner":
> 2020-01-10T20:13:16.3036609Z waiting to lock monitor 0x000000000d540db8 (object 0x00000000ca33e4c8, a sbt.internal.ManagedClassLoader$ZombieClassLoader),
> 2020-01-10T20:13:16.3036689Z which is held by "block-manager-slave-async-thread-pool-81"
> 2020-01-10T20:13:16.3036760Z "block-manager-slave-async-thread-pool-81":
> 2020-01-10T20:13:16.3036837Z waiting to lock monitor 0x000000001fb7e568 (object 0x00000000c1f359f0, a sbt.internal.LayeredClassLoader),
> 2020-01-10T20:13:16.3036912Z which is held by "Spark Context Cleaner"
> 2020-01-10T20:13:16.3036949Z
> 2020-01-10T20:13:16.3037009Z Java stack information for the threads listed above:
> 2020-01-10T20:13:16.3037144Z ===================================================
> 2020-01-10T20:13:16.3037215Z "Thread-3":
> 2020-01-10T20:13:16.3037283Z at org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:273)
> 2020-01-10T20:13:16.3037371Z at org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:121)
> 2020-01-10T20:13:16.3037458Z at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95)
> 2020-01-10T20:13:16.3037528Z "Spark Context Cleaner":
> 2020-01-10T20:13:16.3037590Z at java.lang.ClassLoader.checkCerts(ClassLoader.java:887)
> 2020-01-10T20:13:16.3037672Z - waiting to lock <0x00000000ca33e4c8> (a sbt.internal.ManagedClassLoader$ZombieClassLoader)
> 2020-01-10T20:13:16.3037747Z at java.lang.ClassLoader.preDefineClass(ClassLoader.java:668)
> 2020-01-10T20:13:16.3037821Z at java.lang.ClassLoader.defineClass(ClassLoader.java:761)
> 2020-01-10T20:13:16.3037895Z at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> 2020-01-10T20:13:16.3037982Z at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> 2020-01-10T20:13:16.3038052Z at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> 2020-01-10T20:13:16.3038132Z at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> 2020-01-10T20:13:16.3038199Z at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> 2020-01-10T20:13:16.3038280Z at java.security.AccessController.doPrivileged(Native Method)
> 2020-01-10T20:13:16.3038356Z at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> 2020-01-10T20:13:16.3038435Z at sbt.internal.ManagedClassLoader$ZombieClassLoader.lookupClass(LayeredClassLoaders.scala:336)
> 2020-01-10T20:13:16.3038519Z at sbt.internal.ManagedClassLoader.findClass(LayeredClassLoaders.scala:375)
> 2020-01-10T20:13:16.3038590Z at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 2020-01-10T20:13:16.3038664Z - locked <0x00000000c1f359f0> (a sbt.internal.LayeredClassLoader)
> 2020-01-10T20:13:16.3038739Z at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 2020-01-10T20:13:16.3038819Z at org.apache.spark.storage.BlockManagerMaster.removeShuffle(BlockManagerMaster.scala:138)
> 2020-01-10T20:13:16.3038899Z at org.apache.spark.ContextCleaner.doCleanupShuffle(ContextCleaner.scala:226)
> 2020-01-10T20:13:16.3038996Z at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:192)
> 2020-01-10T20:13:16.3039104Z at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:185)
> 2020-01-10T20:13:16.3039187Z at scala.Option.foreach(Option.scala:257)
> 2020-01-10T20:13:16.3039268Z at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:185)
> 2020-01-10T20:13:16.3039357Z - locked <0x00000000c3d74cd0> (a org.apache.spark.ContextCleaner)
> 2020-01-10T20:13:16.3039438Z at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
> 2020-01-10T20:13:16.3039529Z at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:178)
> 2020-01-10T20:13:16.3039607Z at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73)
> 2020-01-10T20:13:16.3039679Z "block-manager-slave-async-thread-pool-81":
> 2020-01-10T20:13:16.3039814Z at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> 2020-01-10T20:13:16.3039893Z - waiting to lock <0x00000000c1f359f0> (a sbt.internal.LayeredClassLoader)
> 2020-01-10T20:13:16.3039990Z at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
> 2020-01-10T20:13:16.3040063Z - locked <0x00000000ca33e4c8> (a sbt.internal.ManagedClassLoader$ZombieClassLoader)
> 2020-01-10T20:13:16.3040140Z at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 2020-01-10T20:13:16.3040228Z at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply$mcZ$sp(BlockManagerSlaveEndpoint.scala:58)
> 2020-01-10T20:13:16.3040393Z at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply(BlockManagerSlaveEndpoint.scala:57)
> 2020-01-10T20:13:16.3040496Z at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply(BlockManagerSlaveEndpoint.scala:57)
> 2020-01-10T20:13:16.3040600Z at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$1.apply(BlockManagerSlaveEndpoint.scala:86)
> 2020-01-10T20:13:16.3040683Z at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> 2020-01-10T20:13:16.3040767Z at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> 2020-01-10T20:13:16.3040844Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-01-10T20:13:16.3040928Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-01-10T20:13:16.3040999Z at java.lang.Thread.run(Thread.java:748)
> 2020-01-10T20:13:16.3041053Z
> 2020-01-10T20:13:16.3041101Z Found 1 deadlock.
> 2020-01-10T20:13:16.3041130Z
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org