[jira] [Created] (FLINK-36540) Add Support for Hadoop Caller Context when using Flink on yarn

["liang yu (Jira)" <ji...@apache.org>]

liang yu created FLINK-36540:
--------------------------------

             Summary: Add Support for Hadoop Caller Context when using Flink on yarn
                 Key: FLINK-36540
                 URL: https://issues.apache.org/jira/browse/FLINK-36540
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Task
         Environment: Hadoop 3.3.4
Flink 1.14.4
            Reporter: liang yu


*Description:* As HDFS mentioned in HDFS-9184, for a given HDFS operation, it's
very helpful to track which upper level job issues it. But now Flink does not support to set callerContext when trying to operate on Hadoop fileSystem.

 

*implementation:* Add a new HadoopOption which can be set in flink.yaml. By setting this option to true, flink can search for the classpath if we can invoke the method
{code:java}
callerContext.Builder().build().setCurrent(context){code}
 and then set the jobId and TaskName with subtask id so that we can find exactly who did the operation to Hadoop file.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)