You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/01/17 08:22:00 UTC

[jira] [Updated] (FLINK-11378) Allow HadoopRecoverableWriter to write to Hadoop compatible Filesystems.

     [ https://issues.apache.org/jira/browse/FLINK-11378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated FLINK-11378:
-----------------------------------
    Labels: pull-request-available  (was: )

> Allow HadoopRecoverableWriter to write to Hadoop compatible Filesystems.
> ------------------------------------------------------------------------
>
>                 Key: FLINK-11378
>                 URL: https://issues.apache.org/jira/browse/FLINK-11378
>             Project: Flink
>          Issue Type: Improvement
>          Components: FileSystem
>            Reporter: Martijn van de Grift
>            Priority: Major
>              Labels: pull-request-available
>
> At a client we're using Flink jobs to read data from Kafka and writing it to GCS. In earlier versions, we've used `BucketingFileSink` for this, but we want to switch to the newer `StreamingFileSink`.
> Since we're running Flink on Google's DataProc, we're using the Hadoop compatible GCS [connector|https://github.com/GoogleCloudPlatform/bigdata-interop] made by Google. This currently doesn't work on Flink, because Flink checks for a HDFS scheme at 'HadoopRecoverableWriter'.
> We've successfully ran our jobs by creating a custom Flink Distro which has the hdfs scheme check removed.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)