[jira] [Closed] (FLINK-15777) Truncate not working with plugins (as expected)

["Piotr Nowojski (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Piotr Nowojski closed FLINK-15777.
----------------------------------
    Resolution: Cannot Reproduce

The root cause of the issue was missing S3 plugin jar and fallback FileSystemFactory was being used to try to handle the {{s3://path}} writes. {{HadoopRecoverableWriter}} was throwing a confusing error message, that was suggesting wrong hadoop version being used, instead of reporting something like:
{noformat}
"Hadoop Recoverable writer only supports HDFS and cannot be used for writes to s3 schema".
{noformat}

> Truncate not working with plugins (as expected)
> -----------------------------------------------
>
>                 Key: FLINK-15777
>                 URL: https://issues.apache.org/jira/browse/FLINK-15777
>             Project: Flink
>          Issue Type: Bug
>          Components: FileSystems
>    Affects Versions: 1.9.1, 1.10.0, 1.11.0
>            Reporter: Arvid Heise
>            Assignee: Arvid Heise
>            Priority: Critical
>              Labels: pull-request-available
>             Fix For: 1.10.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> FLINK-14170 introduced Hadoop version check to support older versions with limited functionality. However, the fix checks the Hadoop version of the system classloader and not of the plugin (of the actual used filesystem).
> The [version checks|https://github.com/apache/flink/blob/a607bd9dbdda7d9925d6c351ba88d82edce2c571/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L197] will fail when there is a Hadoop version in the classpath of Flink, independent of the bundled version of the filesystem.
> On a recent EMR setup with lingering Hadoop 2.6 libraries, this mean that you cannot write to S3 with StreamingFileSink.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)