[jira] [Commented] (SPARK-38250) Check existence before deleting stagingDir in HadoopMapReduceCommitProtocol

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-38250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494553#comment-17494553 ] 

Apache Spark commented on SPARK-38250:
--------------------------------------

User 'symious' has created a pull request for this issue:
https://github.com/apache/spark/pull/35569

> Check existence before deleting stagingDir in HadoopMapReduceCommitProtocol
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-38250
>                 URL: https://issues.apache.org/jira/browse/SPARK-38250
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.2.1
>            Reporter: Janus Chow
>            Priority: Major
>
> In `commitJob` and `abortJob` of HadoopMapReduceCommitProtocol, the stagingDir was deleted, but for cases like the following example, the stagingDir was not created in the first place.
> If the underLayer FileSystem is HDFS, there won't be any logs, but deleting a nonexistent file with FileSystem.delete would also acquire the WRITE LOCK of NameNode, if such operations are frequent, it would hurt the performance of NN.
> For other FileSystem, like Alluxio, a warning log was thrown as follows.
> {code:java}
> AbstractFileSystem: delete failed: alluxio.exception.FileDoesNotExistException: Path "/test/spark/t10/.spark-staging-3cfe0d7c-3749-44de-9da7-f811a40aa4af" does not exist. {code}
> This ticket is to add an existence check before the actual deleting operation, which could help eliminate the WARN log or not hurt HDFS' performance.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org