[jira] [Created] (HDFS-17577) Add Support for CreateFlag.NO_LOCAL_WRITE in File Creation to Manage Disk Space and Network Load in Labeled YARN Nodes

["liang yu (Jira)" <ji...@apache.org>]

liang yu created HDFS-17577:
-------------------------------

             Summary: Add Support for CreateFlag.NO_LOCAL_WRITE in File Creation to Manage Disk Space and Network Load in Labeled YARN Nodes
                 Key: HDFS-17577
                 URL: https://issues.apache.org/jira/browse/HDFS-17577
             Project: Hadoop HDFS
          Issue Type: New Feature
          Components: dfsclient
            Reporter: liang yu


{*}Description{*}: I am currently using Apache Flink to write files into Hadoop. The Flink application runs on a labeled YARN queue. During operation, it has been observed that the local disks on these labeled nodes get filled up quickly, and the network load is significantly high. This issue arises because Hadoop prioritizes writing files to the local node first, and the number of these labeled nodes is quite limited.

 

{*}Problem{*}: The current behavior leads to inefficient disk space utilization and high network traffic on these few labeled nodes, which could potentially affect the performance and reliability of the application.

 

{*}Implementation{*}: The implementation would involve adding an configuration _dfs.client.write.no_local_write_ to support the {{CreateFlag.NO_LOCAL_WRITE}} during the file creation process in Hadoop's file system APIs. This will provide flexibility to applications like Flink running in labeled queues to opt for non-local writes when necessary.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org