You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Yan Yan (JIRA)" <ji...@apache.org> on 2018/10/10 23:41:00 UTC
[jira] [Created] (FLINK-10526) Hadoop FileSystem not initialized
properly on Yarn
Yan Yan created FLINK-10526:
-------------------------------
Summary: Hadoop FileSystem not initialized properly on Yarn
Key: FLINK-10526
URL: https://issues.apache.org/jira/browse/FLINK-10526
Project: Flink
Issue Type: Bug
Components: YARN
Affects Versions: 1.4.0
Reporter: Yan Yan
When we were bumping Flink from 1.3.2 to 1.4.0, we noticed the HadoopFileSystem no longer reads flinkConfig.
This prevents users to pass in custom HDFS configuration (e.g. core-site.xml) via _fs.hdfs.hadoopconf_.
Specially it is due to the cached _HadoopFsFactory_ is initialized with an dummy Configuration ([[code|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L387]|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L387]), which prevents future _flinkConfig_ getting passed in ([[code|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFsFactory.java#L84]|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFsFactory.java#L84]).
We have verified by calling _FileSystem.initialize(flinkConfig)_ explicitly in _YarnApplicationMasterRunner.run_ solve the issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)