You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "David Klinberg (Jira)" <ji...@apache.org> on 2023/09/05 04:07:00 UTC
[jira] [Resolved] (SPARK-44019) Unable to deserialize broadcasted map statuses when executor decommissioned
[ https://issues.apache.org/jira/browse/SPARK-44019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Klinberg resolved SPARK-44019.
------------------------------------
Resolution: Duplicate
Duplicate with https://issues.apache.org/jira/browse/SPARK-38101
> Unable to deserialize broadcasted map statuses when executor decommissioned
> ---------------------------------------------------------------------------
>
> Key: SPARK-44019
> URL: https://issues.apache.org/jira/browse/SPARK-44019
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 3.1.0, 3.2.0, 3.3.0, 3.4.0
> Environment: Spark version: 3.3.1
> running on Kubernetes
> Reporter: David Klinberg
> Priority: Minor
> Labels: MapOutput, decommision
>
> during examination of graceful executor decommission at high rate of interruptions, jobs occasionally abort on the exception(which isn't reproducd when decommission is disabled):
> {code:java}
> org.apache.spark.shuffle.MetadataFetchFailedException: Unable to deserialize broadcasted map statuses for shuffle xx: java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_yyy_pieceZ of broadcast_yyy{code}
> This exceptions are reproduced from reducer tasks while multiple decommission are in progress or finished which most of the time causes the entire stage to fail due to its highly reproduced.
> This seems to be related to `updateMapOuput` broadcast invalidation which destroys and creates those broadcast variables on each shuffle migration([reference|https://github.com/apache/spark/blob/v3.1.0-rc1/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L134]).
> *Example sequence of events from driver:*
> {code:java}
> 2023-06-03 18:41:17:798 INFO TorrentBroadcast:61 Destroying Broadcast(171) (from updateMapOutput at BlockManagerMasterEndpoint.scala:639)
> ...
> 2023-06-03 18:41:17:811 INFO BlockManagerInfo:61 Removed broadcast_171_piece0 on xxx_host_xxx:7079 in memory (size: 4.0 MiB, free: 10.4 GiB)
> ...
> 2023-06-03 18:41:17:841 WARN TaskSetManager:73 Lost task 6.0 in stage 124.2 (TID xxxxx) (x.x.x.x executor xxxx): FetchFailed(null, shuffleId=xx, mapIndex=-1, mapId=-1, reduceId=-1, message=
> org.apache.spark.shuffle.MetadataFetchFailedException: Unable to deserialize broadcasted map statuses for shuffle xx: java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_171_piece0 of broadcast_171{{}}{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org