You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Eric Yang (Jira)" <ji...@apache.org> on 2026/05/24 17:37:00 UTC
[jira] [Updated] (SPARK-57041) Fix deadlock between waitForSubqueries and lazy val initialization in subquery thread
[ https://issues.apache.org/jira/browse/SPARK-57041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated SPARK-57041:
------------------------------
Description:
A deadlock can occur between a broadcast-exchange thread and a subquery thread:
- Thread A (broadcast-exchange): enters SparkPlan.waitForSubqueries(), which acquires this.synchronized, then blocks waiting for a subquery future.
- Thread B (subquery-0): while executing the subquery under AQE, onUpdatePlan calls SparkPlanInfo.fromSparkPlan → FileSourceScanLike.metadata (a Scala lazy val). Scala compiles lazy val to this.synchronized, so Thread B blocks trying to acquire the same lock Thread A holds.
subquery thread:
{code:java}
"subquery-0" prio=5 tid=486 BLOCKED deamon
native=false, suspended=false, block=12, wait=0
lock=org.apache.spark.sql.execution.FileSourceScanExec@6a25b17b owned by broadcast-exchange-12-89594345-aeb3-4343-aed7-1e6399b9f2e9 (489), cpu=114, user=80
app//org.apache.spark.sql.execution.FileSourceScanExec.metadata$lzycompute(DataSourceScanExec.scala:729)
app//org.apache.spark.sql.execution.FileSourceScanExec.metadata(DataSourceScanExec.scala:729)
app//org.apache.spark.sql.execution.SparkPlanInfo$.fromSparkPlan(SparkPlanInfo.scala:93) {code}
broadcast-exchange thread:
{code:java}
"broadcast-exchange-12-89594345-aeb3-4343-aed7-1e6399b9f2e9" prio=5 tid=489 WAITING deamon
native=false, suspended=false, block=0, wait=1
lock=java.util.concurrent.FutureTask@282caa50 owned by null (-1), cpu=14, user=10
java.base@17.0.14/jdk.internal.misc.Unsafe.park(Native Method)
java.base@17.0.14/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
java.base@17.0.14/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:447)
java.base@17.0.14/java.util.concurrent.FutureTask.get(FutureTask.java:190)
app//org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:331)
app//org.apache.spark.sql.execution.SubqueryExec.executeCollect(basicPhysicalOperators.scala:925)
app//org.apache.spark.sql.execution.InSubqueryExec.updateResult(subquery.scala:131)
{code}
was:
A deadlock can occur between a broadcast-exchange thread and a subquery thread:
- Thread A (broadcast-exchange): enters SparkPlan.waitForSubqueries(), which acquires this.synchronized, then blocks waiting for a subquery future.
- Thread B (subquery-0): while executing the subquery under AQE, onUpdatePlan calls SparkPlanInfo.fromSparkPlan → FileSourceScanLike.metadata (a Scala lazy val). Scala compiles lazy val to this.synchronized, so Thread B blocks trying to acquire the same lock Thread A holds.
> Fix deadlock between waitForSubqueries and lazy val initialization in subquery thread
> -------------------------------------------------------------------------------------
>
> Key: SPARK-57041
> URL: https://issues.apache.org/jira/browse/SPARK-57041
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.1
> Reporter: Eric Yang
> Priority: Major
>
> A deadlock can occur between a broadcast-exchange thread and a subquery thread:
> - Thread A (broadcast-exchange): enters SparkPlan.waitForSubqueries(), which acquires this.synchronized, then blocks waiting for a subquery future.
> - Thread B (subquery-0): while executing the subquery under AQE, onUpdatePlan calls SparkPlanInfo.fromSparkPlan → FileSourceScanLike.metadata (a Scala lazy val). Scala compiles lazy val to this.synchronized, so Thread B blocks trying to acquire the same lock Thread A holds.
> subquery thread:
> {code:java}
> "subquery-0" prio=5 tid=486 BLOCKED deamon
> native=false, suspended=false, block=12, wait=0
> lock=org.apache.spark.sql.execution.FileSourceScanExec@6a25b17b owned by broadcast-exchange-12-89594345-aeb3-4343-aed7-1e6399b9f2e9 (489), cpu=114, user=80
> app//org.apache.spark.sql.execution.FileSourceScanExec.metadata$lzycompute(DataSourceScanExec.scala:729)
> app//org.apache.spark.sql.execution.FileSourceScanExec.metadata(DataSourceScanExec.scala:729)
> app//org.apache.spark.sql.execution.SparkPlanInfo$.fromSparkPlan(SparkPlanInfo.scala:93) {code}
> broadcast-exchange thread:
> {code:java}
> "broadcast-exchange-12-89594345-aeb3-4343-aed7-1e6399b9f2e9" prio=5 tid=489 WAITING deamon
> native=false, suspended=false, block=0, wait=1
> lock=java.util.concurrent.FutureTask@282caa50 owned by null (-1), cpu=14, user=10
> java.base@17.0.14/jdk.internal.misc.Unsafe.park(Native Method)
> java.base@17.0.14/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
> java.base@17.0.14/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:447)
> java.base@17.0.14/java.util.concurrent.FutureTask.get(FutureTask.java:190)
> app//org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:331)
> app//org.apache.spark.sql.execution.SubqueryExec.executeCollect(basicPhysicalOperators.scala:925)
> app//org.apache.spark.sql.execution.InSubqueryExec.updateResult(subquery.scala:131)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org