[jira] [Created] (FLINK-17253) Support writing to viewfs for hadoop versions < 2.7 when using BulkFormatBuilder in StreamingFileSink

["chaganti spurthi (Jira)" <ji...@apache.org>]

chaganti spurthi created FLINK-17253:
----------------------------------------

             Summary: Support writing to viewfs for hadoop versions < 2.7 when using BulkFormatBuilder in StreamingFileSink
                 Key: FLINK-17253
                 URL: https://issues.apache.org/jira/browse/FLINK-17253
             Project: Flink
          Issue Type: Bug
          Components: Connectors / FileSystem
    Affects Versions: 1.9.0, 1.8.3, 1.8.2, 1.8.1, 1.8.0
            Reporter: chaganti spurthi


FLINK-14170 introduced Hadoop version check to support older hadoop versions. However the check only included "hdfs" scheme but not "viewfs". We are using StreamingFileSink to write data to our federated hadoop cluster with cdh-2.6 hadoop version and we are hit with
{code:java}
java.lang.UnsupportedOperationException: Recoverable writers on Hadoop are only supported for HDFS and for Hadoop version 2.7 or newer at org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriter.<init>(HadoopRecoverableWriter.java:61) at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202) at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69) at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$BulkFormatBuilder.createBuckets(StreamingFileSink.java:317) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281) at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:901) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:415) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at java.lang.Thread.run(Thread.java:748)
{code}
The change is add viewfs to the scheme check. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)