[jira] [Created] (FLINK-10203) HadoopRecoverableWriter does not support Hadoop

["Artsem Semianenka (JIRA)" <ji...@apache.org>]

Artsem Semianenka created FLINK-10203:
-----------------------------------------

             Summary: HadoopRecoverableWriter does not support Hadoop 
                 Key: FLINK-10203
                 URL: https://issues.apache.org/jira/browse/FLINK-10203
             Project: Flink
          Issue Type: Bug
          Components: DataStream API, filesystem-connector
    Affects Versions: 1.6.0, 1.6.1, 1.7.0
            Reporter: Artsem Semianenka


New StreamingFileSink ( introduced in 1.6 Flink version ) use HadoopRecoverableFsDataOutputStream wrapper to write data in HDFS.

HadoopRecoverableFsDataOutputStream is an wrapper on FSDataOutputStream to have an ability to restore from certain point of file after failure and continue write data. To achive this recover functionality the HadoopRecoverableFsDataOutputStream use "truncate" method which was introduced only in Hadoop 2.7 .

Unfortently there are a few official Hadoop distibutives which latast version still use Hadoop 2.6 (This distibutives: Cloudera, Pivotal HD ). As the result Flink can't work with this distibutives.

Flink declares that supported Hadoop from version 2.4.0 upwards (https://ci.apache.org/projects/flink/flink-docs-release-1.6/start/building.html#hadoop-versions)

I guess we should emulate the functionality of "truncate" method for older Hadoop versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)