[jira] [Comment Edited] (FLINK-9683) inconsistent behaviors when setting historyserver.archive.fs.dir

["Jiang Xin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271033#comment-17271033 ] 

Jiang Xin edited comment on FLINK-9683 at 1/25/21, 2:36 AM:
------------------------------------------------------------

I believe this is expected behavior. fs.default-scheme is not designed for web server and other "not flink file". Therefore we use WebMonitorUtils#validateAndNormalizeUri() to check the scheme.


was (Author: jiang xin):
I believe this is expected behavior. fs.default-scheme is not designed for web server and other "not flink file". Therefor we use WebMonitorUtils#validateAndNormalizeUri() to check the scheme.

> inconsistent behaviors when setting historyserver.archive.fs.dir
> ----------------------------------------------------------------
>
>                 Key: FLINK-9683
>                 URL: https://issues.apache.org/jira/browse/FLINK-9683
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Coordination
>    Affects Versions: 1.4.2
>            Reporter: Ethan Li
>            Priority: Major
>              Labels: starter
>
> I am using release-1.4.2,
>  
> With fs.default-scheme and  fs.hdfs.hadoopconf set correctly, 
> when setting 
> {code:java}
> historyserver.archive.fs.dir: /tmp/flink/cluster-name/jmarchive
> {code}
> I am seeing
> {code:java}
> 2018-06-27 18:51:12,692 WARN org.apache.flink.runtime.webmonitor.history.HistoryServer - Failed to create Path or FileSystem for directory '/tmp/flink/cluster-name/jmarchive'. Directory will not be monitored.
> java.lang.IllegalArgumentException: The scheme (hdfs://, file://, etc) is null. Please specify the file system scheme explicitly in the URI.
> at org.apache.flink.runtime.webmonitor.WebMonitorUtils.validateAndNormalizeUri(WebMonitorUtils.java:300)
> at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:168)
> at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)
> at org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:113)
> at org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:110)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.runtime.webmonitor.history.HistoryServer.main(HistoryServer.java:110)
> {code}
>  
> And then if I set 
> {code:java}
> historyserver.archive.fs.dir: hdfs:///tmp/flink/cluster-name/jmarchive{code}
> I am seeing:
>  
> {code:java}
> java.io.IOException: The given file system URI (hdfs:///tmp/flink/cluster-name/jmarchive) did not describe the authority (like for example HDFS NameNode address/port or S3 host). The attempt to use a configured default authority failed: Hadoop configuration for default file system ('fs.default.name' or 'fs.defaultFS') contains no valid authority component (like hdfs namenode, S3 host, etc)
> at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
> at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
> at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:169)
> at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)
> {code}
> The only way it works is to provide the full path of hdfs like:
> {code:java}
> #historyserver.archive.fs.dir: hdfs://<hdfs-cluster>/tmp/flink/cluster-name/jmarchive
> {code}
>  
> Above situations are because there are two parts of code treating "scheme" differently. 
> https://github.com/apache/flink/blob/release-1.4.2/flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorUtils.java#L299-L302
> https://github.com/apache/flink/blob/release-1.4.2/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L335-L338
> I believe the first case should be supported if users have set fs.default-scheme 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)