You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/11/13 19:37:00 UTC
[jira] [Assigned] (SPARK-26042)
KafkaContinuousSourceTopicDeletionSuite may hang forever
[ https://issues.apache.org/jira/browse/SPARK-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-26042:
------------------------------------
Assignee: Shixiong Zhu (was: Apache Spark)
> KafkaContinuousSourceTopicDeletionSuite may hang forever
> --------------------------------------------------------
>
> Key: SPARK-26042
> URL: https://issues.apache.org/jira/browse/SPARK-26042
> Project: Spark
> Issue Type: Test
> Components: Structured Streaming, Tests
> Affects Versions: 2.4.0
> Reporter: Shixiong Zhu
> Assignee: Shixiong Zhu
> Priority: Major
>
> Saw the following thread dump in some build:
> {code}
> "stream execution thread for [id = 1c13482e-1edf-4b5c-b63a-d652738c8a48, runId = 10667ce9-7eac-4cef-a525-f1bd08eb50f1]" #4406 daemon prio=5 os_prio=0 tid=0x00007fab1d3c5000 nid=0x7f4b waiting on condition [0x00007fa96efcb000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000000070a904cf8> (a scala.concurrent.impl.Promise$CompletionLatch)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> ...
> at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:180)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:131)
> at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:109)
> - locked <0x000000070a913ee8> (a org.apache.spark.sql.execution.streaming.IncrementalExecution)
> at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:109)
> at org.apache.spark.sql.execution.streaming.continuous.ContinuousExecution$$anonfun$runContinuous$3$$anonfun$apply$1.apply(ContinuousExecution.scala:270)
> at org.apache.spark.sql.execution.streaming.continuous.ContinuousExecution$$anonfun$runContinuous$3$$anonfun$apply$1.apply(ContinuousExecution.scala:270)
> ,,,
> "pool-1-thread-1-ScalaTest-running-KafkaContinuousSourceTopicDeletionSuite" #20 prio=5 os_prio=0 tid=0x00007fabc4e78800 nid=0x23be waiting for monitor entry [0x00007fab3dbff000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:100)
> - waiting to lock <0x000000070a913ee8> (a org.apache.spark.sql.execution.streaming.IncrementalExecution)
> at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:100)
> at org.apache.spark.sql.kafka010.KafkaContinuousSourceTopicDeletionSuite$$anonfun$3$$anonfun$apply$mcV$sp$12$$anonfun$apply$15.apply(KafkaContinuousSourceSuite.scala:210)
> at org.apache.spark.sql.kafka010.KafkaContinuousSourceTopicDeletionSuite$$anonfun$3$$anonfun$apply$mcV$sp$12$$anonfun$apply$15.apply(KafkaContinuousSourceSuite.scala:209)
> ...
> {code}
> It hung forever because the test main thread was trying to access `executedPlan` but the lock was held by the streaming thread.
> This is a pretty common issue when using lazy vals as all lazy vals share the same lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org