[jira] [Commented] (FLINK-15777) Truncate not working with plugins (as expected)

["Kostas Kloudas (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024385#comment-17024385 ] 

Kostas Kloudas commented on FLINK-15777:
----------------------------------------

[~arvid heise] well the checks existed since the introduction of the {{StreamingFileSink}}. The commit mentioned in the description just moved them from the task initialisation, further "down" the task runtime, so this is a regression compared to previous Flink versions not related to [FLINK-14170|https://issues.apache.org/jira/browse/FLINK-14170], but probably to the new plugin mechanism integration with the StreamingFileSink.

> Truncate not working with plugins (as expected)
> -----------------------------------------------
>
>                 Key: FLINK-15777
>                 URL: https://issues.apache.org/jira/browse/FLINK-15777
>             Project: Flink
>          Issue Type: Bug
>          Components: FileSystems
>    Affects Versions: 1.10.0, 1.11.0
>            Reporter: Arvid Heise
>            Priority: Critical
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> FLINK-14170 introduced Hadoop version check to support older versions with limited functionality. However, the fix checks the Hadoop version of the system classloader and not of the plugin (of the actual used filesystem).
> The [version checks|https://github.com/apache/flink/blob/a607bd9dbdda7d9925d6c351ba88d82edce2c571/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L197] will fail when there is a Hadoop version in the classpath of Flink, independent of the bundled version of the filesystem.
> On a recent EMR setup with lingering Hadoop 2.6 libraries, this mean that you cannot write to S3 with StreamingFileSink.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)