You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "CHANHAE OH (Jira)" <ji...@apache.org> on 2026/02/09 02:23:00 UTC
[jira] [Updated] (FLINK-39033) File Connector does not support WebHDFS protocol
[ https://issues.apache.org/jira/browse/FLINK-39033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
CHANHAE OH updated FLINK-39033:
-------------------------------
Attachment: (was: image-2026-02-05-14-59-51-374.png)
> File Connector does not support WebHDFS protocol
> ------------------------------------------------
>
> Key: FLINK-39033
> URL: https://issues.apache.org/jira/browse/FLINK-39033
> Project: Flink
> Issue Type: Bug
> Components: API / Core
> Affects Versions: 2.0.1
> Reporter: CHANHAE OH
> Priority: Minor
> Fix For: 2.0.1
>
>
> We encountered an issue where the File Connector does not support the WebHDFS protocol.
>
> >flink/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableWriter.java
>
> {code:java}
> protected void checkSupportedFSSchemes(org.apache.hadoop.fs.FileSystem fs) {
> // This writer is only supported on a subset of file systems
> if (!("hdfs".equalsIgnoreCase(fs.getScheme())
> || "viewfs".equalsIgnoreCase(fs.getScheme()))) {
> throw new UnsupportedOperationException(
> "Recoverable writers on Hadoop are only supported for HDFS");
> }
> // Part of functionality depends on specific versions. We check these schemes and versions
> // eagerly for
> // better error messages.
> if (!HadoopUtils.isMinHadoopVersion(2, 7)) {
> LOG.warn(
> "WARNING: You are running on hadoop version "
> + VersionInfo.getVersion()
> + "."
> + " If your RollingPolicy does not roll on every checkpoint/savepoint, the StreamingFileSink will throw an exception upon recovery.");
> }
> } {code}
>
>
> Interestingly, when using the Iceberg Connector in the same environment, data can be
> successfully written to WebHDFS paths without any issues. This suggests that WebHDFS
> itself is a supported protocol, and upon further investigation, it may be possible for
> the File Connector to support WebHDFS as well.
>
> Would it be possible for me to submit a PR to improve this? If so, could you provide
> any guidance on what areas I should examine?
> Thank you!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)