[jira] [Closed] (FLINK-20682) Add configuration options related to hadoop

["Matthias (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias closed FLINK-20682.
----------------------------
    Resolution: Duplicate

Hi [~jiang7chengzitc], thanks for bringing this up. FLINK-16005 covers this issue already. I'm going to close this issue in favor of FLINK-16005 and assign you to it.

> Add configuration options related to hadoop
> -------------------------------------------
>
>                 Key: FLINK-20682
>                 URL: https://issues.apache.org/jira/browse/FLINK-20682
>             Project: Flink
>          Issue Type: Improvement
>          Components: Deployment / YARN
>    Affects Versions: 1.12.0
>            Reporter: Ruguo Yu
>            Priority: Major
>              Labels: pull-requests-available
>             Fix For: 1.13.0, 1.12.1
>
>
> Current, we submit flink job to yarn with run-application target and need to specify some configuration related to hadoop, because we use distributed filesystem similar to Ali oss to storage resources, in this case, we will pass special configuration option and set them to hadoopConfiguration.
> In order to solve such problems, we can provide a configuration option prefixed with "flink.hadoop."(such as -Dflink.hadoop.xxx=yyy), and then take it into HadoopConfiguration.
> A simple implementation code is as follows:
> {code:java}
> module: flink-filesystems/flink-hadoop-fs
> class: org.apache.flink.runtime.util.HadoopUtils
> //代码占位符
> public static Configuration getHadoopConfiguration(org.apache.flink.configuration.Configuration flinkConfiguration) {
>    ......
>    // Copy any "flink.hadoop.xxx=yyy" flink configuration to hadoop configuration as "xxx=yyy"
>    for (String key : flinkConfiguration.keySet()) {
>       if (key.startsWith("flink.hadoop.")) {
>          result.set(key.substring("flink.hadoop.".length()),
>             flinkConfiguration.getString(key, null));
>       }
>    }
>    return result;
> }{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)