You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Arvid Heise (Jira)" <ji...@apache.org> on 2020/01/27 13:36:00 UTC

[jira] [Created] (FLINK-15777) Truncate not working with plugins (as expected)

Arvid Heise created FLINK-15777:
-----------------------------------

             Summary: Truncate not working with plugins (as expected)
                 Key: FLINK-15777
                 URL: https://issues.apache.org/jira/browse/FLINK-15777
             Project: Flink
          Issue Type: Bug
          Components: FileSystems
    Affects Versions: 1.10.0, 1.11.0
            Reporter: Arvid Heise


FLINK-14170 introduced Hadoop version check to support older versions with limited functionality. However, the fix checks the Hadoop version of the system classloader and not of the plugin (of the actual used filesystem).

The [version checks|https://github.com/apache/flink/blob/a607bd9dbdda7d9925d6c351ba88d82edce2c571/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L197] will fail when there is a Hadoop version in the classpath of Flink, independent of the bundled version of the filesystem.

On a recent EMR setup with lingering Hadoop 2.6 libraries, this mean that you cannot write to S3 with StreamingFileSink.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)