You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "wang-zhun (Jira)" <ji...@apache.org> on 2021/11/08 07:32:00 UTC
[jira] [Created] (SPARK-37239) Avoid unnecessary `setReplication`
in Yarn mode
wang-zhun created SPARK-37239:
---------------------------------
Summary: Avoid unnecessary `setReplication` in Yarn mode
Key: SPARK-37239
URL: https://issues.apache.org/jira/browse/SPARK-37239
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 3.1.2
Reporter: wang-zhun
We found a large number of replication logs in hdfs server
```
2021-11-04,17:22:13,065 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: Replication remains unchanged at 3 for xxx/.sparkStaging/application_1635470728320_1144379/__spark_libs__303253482044663796.zip
2021-11-04,17:22:13,069 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: Replication remains unchanged at 3 for xxx/.sparkStaging/application_1635470728320_1144383/__spark_libs__4747402134564993861.zip
2021-11-04,17:22:13,070 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: Replication remains unchanged at 3 for xxx/.sparkStaging/application_1635470728320_1144373/__spark_libs__4377509773730188331.zip
```
https://github.com/apache/hadoop/blob/6f7b965808f71f44e2617c50d366a6375fdfbbfa/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L2439
`setReplication` needs to acquire write lock, we should reduce this unnecessary operation
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org