You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wu Yi (Jira)" <ji...@apache.org> on 2026/03/10 03:45:00 UTC
[jira] [Assigned] (SPARK-54796) NPE caused by race condition between Executor initialization and shuffle migration
[ https://issues.apache.org/jira/browse/SPARK-54796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wu Yi reassigned SPARK-54796:
-----------------------------
Assignee: Tengfei Huang
> NPE caused by race condition between Executor initialization and shuffle migration
> ----------------------------------------------------------------------------------
>
> Key: SPARK-54796
> URL: https://issues.apache.org/jira/browse/SPARK-54796
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.1.0
> Reporter: Tengfei Huang
> Assignee: Tengfei Huang
> Priority: Major
> Labels: pull-request-available
>
> When there is executor decommission and shuffles data need to be migrated, will ask master to `getPeers` as the target for shuffle data migration.
> And the blockManager would be a candidate right after the blockManager get `initialized` and registered to master, while at that time the `Executor` could be not fully initialized, and the `SparkEnv.shuffleManager` could be null.
> In Executor, we initialize blockManager earlier than shuffleManager.
> [https://github.com/apache/spark/blob/d0cbad56a10502a1c931d5967beeae2369f6fa15/core/src/main/scala/org/apache/spark/executor/Executor.scala#L163]
> [https://github.com/apache/spark/blob/d0cbad56a10502a1c931d5967beeae2369f6fa15/core/src/main/scala/org/apache/spark/executor/Executor.scala#L371]
> While handling the shuffle migration requests, the `lazy shuffleManager` in `BlockManager` would be initialized as `null`. Then later operations on `shuffleManager` would lead to `NullPointerException`.
> Error logs:
> ```
> 2025-12-22 10:00:06 ERROR NettyBlockTransferService:110 - Error while uploading shuffle_0_0_0
> .data as stream
> java.lang.RuntimeException: java.lang.NullPointerException: Cannot invoke "org.apache.spark.shuffle.ShuffleManager.shuffleBlockResolver()" because the return value of "org.apache.spark.storage.BlockManager.shuffleManager()" is null
> at org.apache.spark.storage.BlockManager.migratableResolver$lzycompute(BlockManager.scala:314)
> at org.apache.spark.storage.BlockManager.migratableResolver(BlockManager.scala:313)
> at org.apache.spark.storage.BlockManager.putBlockDataAsStream(BlockManager.scala:777)
> at org.apache.spark.network.netty.NettyBlockRpcServer.receiveStream(NettyBlockRpcServer.scala:184)
> at org.apache.spark.network.server.TransportRequestHandler.processStreamUpload(TransportRequestHandler.java:208)
> at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:117)
> at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:143)
> at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:55)
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:356)
> at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:293)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:354)
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org