[jira] [Created] (FLINK-9683) inconsistent behaviors when setting historyserver.archive.fs.dir

["Ethan Li (JIRA)" <ji...@apache.org>]

Ethan Li created FLINK-9683:
-------------------------------

             Summary: inconsistent behaviors when setting historyserver.archive.fs.dir
                 Key: FLINK-9683
                 URL: https://issues.apache.org/jira/browse/FLINK-9683
             Project: Flink
          Issue Type: Bug
    Affects Versions: 1.4.2
            Reporter: Ethan Li


I am using release-1.4.2,

 

With fs.default-scheme and  fs.hdfs.hadoopconf set correctly, 

when setting 
{code:java}
historyserver.archive.fs.dir: /tmp/flink/cluster-name/jmarchive
{code}
I am seeing
{code:java}
2018-06-27 18:51:12,692 WARN org.apache.flink.runtime.webmonitor.history.HistoryServer - Failed to create Path or FileSystem for directory '/tmp/flink/cluster-name/jmarchive'. Directory will not be monitored.
java.lang.IllegalArgumentException: The scheme (hdfs://, file://, etc) is null. Please specify the file system scheme explicitly in the URI.
at org.apache.flink.runtime.webmonitor.WebMonitorUtils.validateAndNormalizeUri(WebMonitorUtils.java:300)
at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:168)
at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)
at org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:113)
at org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.runtime.webmonitor.history.HistoryServer.main(HistoryServer.java:110)
{code}
 

And then if I set 
{code:java}
historyserver.archive.fs.dir: hdfs:///tmp/flink/cluster-name/jmarchive{code}
I am seeing:

 
{code:java}
java.io.IOException: The given file system URI (hdfs:///tmp/flink/cluster-name/jmarchive) did not describe the authority (like for example HDFS NameNode address/port or S3 host). The attempt to use a configured default authority failed: Hadoop configuration for default file system ('fs.default.name' or 'fs.defaultFS') contains no valid authority component (like hdfs namenode, S3 host, etc)
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:169)
at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)
{code}
The only way it works is to provide the full path of hdfs like:
{code:java}
#historyserver.archive.fs.dir: hdfs://<hdfs-cluster>/tmp/flink/cluster-name/jmarchive
{code}
 

Above situations are because there are two parts of code treating "scheme" differently. 

https://github.com/apache/flink/blob/release-1.4.2/flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorUtils.java#L299-L302

https://github.com/apache/flink/blob/release-1.4.2/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L335-L338

I believe the first case should be supported if users have set fs.default-scheme 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)