You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Janus Chow (Jira)" <ji...@apache.org> on 2022/02/18 10:56:00 UTC
[jira] [Created] (SPARK-38250) Check existence before deleting stagingDir in HadoopMapReduceCommitProtocol
Janus Chow created SPARK-38250:
----------------------------------
Summary: Check existence before deleting stagingDir in HadoopMapReduceCommitProtocol
Key: SPARK-38250
URL: https://issues.apache.org/jira/browse/SPARK-38250
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.2.1
Reporter: Janus Chow
In `commitJob` and `abortJob` of HadoopMapReduceCommitProtocol, the stagingDir was deleted, but for cases like the following example, the stagingDir was not created in the first place.
If the underLayer FileSystem is HDFS, there won't be any logs, but deleting a nonexistent file with FileSystem.delete would also acquire the WRITE LOCK of NameNode, if such operations are frequent, it would hurt the performance of NN.
For other FileSystem, like Alluxio, a warning log was thrown as follows.
{code:java}
AbstractFileSystem: delete failed: alluxio.exception.FileDoesNotExistException: Path "/test/spark/t10/.spark-staging-3cfe0d7c-3749-44de-9da7-f811a40aa4af" does not exist. {code}
This ticket is to add an existence check before the actual deleting operation, which could help eliminate the WARN log or not hurt HDFS' performance.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org