You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/03/06 14:24:00 UTC
[jira] [Updated] (FLINK-11838) Create RecoverableWriter for GCS
[ https://issues.apache.org/jira/browse/FLINK-11838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated FLINK-11838:
-----------------------------------
Labels: pull-request-available (was: )
> Create RecoverableWriter for GCS
> --------------------------------
>
> Key: FLINK-11838
> URL: https://issues.apache.org/jira/browse/FLINK-11838
> Project: Flink
> Issue Type: Improvement
> Components: FileSystems
> Affects Versions: 1.8.0
> Reporter: Fokko Driesprong
> Assignee: Fokko Driesprong
> Priority: Major
> Labels: pull-request-available
>
> GCS supports the resumable upload which we can use to create a Recoverable writer similar to the S3 implementation:
> https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload
> After using the Hadoop compatible interface: https://github.com/apache/flink/pull/7519
> We've noticed that the current implementation relies heavily on the renaming of the files on the commit:
> https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L233-L259
> This is suboptimal on an object store such as GCS. Therefore we would like to implement a more GCS native RecoverableWriter
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)