You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Congxian Qiu(klion26) (Jira)" <ji...@apache.org> on 2019/10/08 04:14:00 UTC

[jira] [Created] (FLINK-14340) Specify an unique DFSClient name for Hadoop FileSystem

Congxian Qiu(klion26) created FLINK-14340:
---------------------------------------------

             Summary: Specify an unique DFSClient name for Hadoop FileSystem
                 Key: FLINK-14340
                 URL: https://issues.apache.org/jira/browse/FLINK-14340
             Project: Flink
          Issue Type: Improvement
          Components: FileSystems
            Reporter: Congxian Qiu(klion26)
             Fix For: 1.10.0


Currently, when Flink read/write to HDFS, we do not set the DFSClient name for all the connections, so we can’t distinguish the connections, and can’t find the specific Job or TM quickly.

This issue wants to add the {{container_id}} as a unique name when init Hadoop File System, so we can easily distinguish the connections belongs to which Job/TM.

 

Core changes is add a line such as below in {{org.apache.flink.runtime.fs.hdfs.HadoopFsFactory#create}}

 
{code:java}
hadoopConfig.set(“mapreduce.task.attempt.id”, System.getenv().getOrDefault(CONTAINER_KEY_IN_ENV, DEFAULT_CONTAINER_ID));{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)