You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Gabor Somogyi (Jira)" <ji...@apache.org> on 2022/10/25 10:22:00 UTC
[jira] [Comment Edited] (FLINK-29754) HadoopConfigLoader should consider Hadoop configuration files
[ https://issues.apache.org/jira/browse/FLINK-29754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17623734#comment-17623734 ]
Gabor Somogyi edited comment on FLINK-29754 at 10/25/22 10:21 AM:
------------------------------------------------------------------
I see the intention. Now it really hits-in that we have 6 different ways how we load Hadoop config. 2 from 6 are in the `flink-hadoop-fs` module, please see [this|https://github.com/apache/flink/blob/0e612856772d5f469c7d4a4fff90a58b6e0f5578/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L53-L54].
was (Author: gaborgsomogyi):
I see the intention. Now it really hits-in that we have 6 different ways how we load Hadoop config. 2 from 6 are in the `flink-hadoop-fs` module, plese see [this|https://github.com/apache/flink/blob/0e612856772d5f469c7d4a4fff90a58b6e0f5578/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L53-L54].
> HadoopConfigLoader should consider Hadoop configuration files
> -------------------------------------------------------------
>
> Key: FLINK-29754
> URL: https://issues.apache.org/jira/browse/FLINK-29754
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Reporter: Peter Vary
> Priority: Major
>
> Currently [HadoopConfigLoader|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopConfigLoader.java] considers Hadoop configurations on the classpath, but does not consider Hadoop configuration files which are set in another way.
> So if the Hadoop configuration is set through the {{HADOOP_CONF_DIR}} environment variable, then the configuration loaded by the HadoopConfigLoader will not contain the values set there.
> This can cause unexpected behaviour when setting checkpoint / savepoint dirs on S3, and the specific S3 configurations are set in the Hadoop configuration files
--
This message was sent by Atlassian Jira
(v8.20.10#820010)