You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Rohit Agrawal (Jira)" <ji...@apache.org> on 2020/01/10 22:04:00 UTC
[jira] [Created] (SPARK-30488) Deadlock between
block-manager-slave-async-thread-pool and spark context cleaner
Rohit Agrawal created SPARK-30488:
-------------------------------------
Summary: Deadlock between block-manager-slave-async-thread-pool and spark context cleaner
Key: SPARK-30488
URL: https://issues.apache.org/jira/browse/SPARK-30488
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.3
Reporter: Rohit Agrawal
Deadlock happens while cleaning up the spark context. Here is the full thread dump:
at org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:121)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95)
"Spark Context Cleaner":
at java.lang.ClassLoader.checkCerts(ClassLoader.java:887)
- waiting to lock <0x00000000ca33e4c8> (a sbt.internal.ManagedClassLoader$ZombieClassLoader)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:668)
at java.lang.ClassLoader.defineClass(ClassLoader.java:761)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at sbt.internal.ManagedClassLoader$ZombieClassLoader.lookupClass(LayeredClassLoaders.scala:336)
at sbt.internal.ManagedClassLoader.findClass(LayeredClassLoaders.scala:375)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
- locked <0x00000000c1f359f0> (a sbt.internal.LayeredClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.storage.BlockManagerMaster.removeShuffle(BlockManagerMaster.scala:138)
at org.apache.spark.ContextCleaner.doCleanupShuffle(ContextCleaner.scala:226)
at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:192)
at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:185)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:185)
- locked <0x00000000c3d74cd0> (a org.apache.spark.ContextCleaner)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:178)
at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73)
"block-manager-slave-async-thread-pool-81":
at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
- waiting to lock <0x00000000c1f359f0> (a sbt.internal.LayeredClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
- locked <0x00000000ca33e4c8> (a sbt.internal.ManagedClassLoader$ZombieClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply$mcZ$sp(BlockManagerSlaveEndpoint.scala:58)
at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply(BlockManagerSlaveEndpoint.scala:57)
at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply(BlockManagerSlaveEndpoint.scala:57)
at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$1.apply(BlockManagerSlaveEndpoint.scala:86)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org