You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/08/31 09:21:00 UTC
[jira] [Commented] (SPARK-44632) DiskBlockManager should check and be able to handle stale directories
[ https://issues.apache.org/jira/browse/SPARK-44632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760816#comment-17760816 ]
ASF GitHub Bot commented on SPARK-44632:
----------------------------------------
User 'yaooqinn' has created a pull request for this issue:
https://github.com/apache/spark/pull/42287
> DiskBlockManager should check and be able to handle stale directories
> ---------------------------------------------------------------------
>
> Key: SPARK-44632
> URL: https://issues.apache.org/jira/browse/SPARK-44632
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.4.1, 3.5.0
> Reporter: Kent Yao
> Priority: Major
>
> The subDir in the memory cache could be stale, for example, after a damaged disk repair or replacement. This dir could be accessed subsequently by others. Especially, `filename` generated by `RDDBlockId` is unchanged between task reties, so it probably attempts to access the same subDir repeatedly. Therefore, it is necessary to check if the subDir exists. If it is stale and the hardware has been recovered without data and directories, we will recreate the subDir to prevent FileNotFoundException during writing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org