You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Robert Metzger (Jira)" <ji...@apache.org> on 2021/01/25 10:41:00 UTC
[jira] [Commented] (FLINK-19481) Add support for a flink native GCS
FileSystem
[ https://issues.apache.org/jira/browse/FLINK-19481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271241#comment-17271241 ]
Robert Metzger commented on FLINK-19481:
----------------------------------------
I'm generally +1 on adding this to Flink. I've seen quite a few people on the ML that seem to use GCS.
Once we have somebody who's willing to drive this, let's see how that is, and whether we need an ML discussion (if there's a committer strongly supporting this, then I don't think we need a ML discussion).
> Add support for a flink native GCS FileSystem
> ---------------------------------------------
>
> Key: FLINK-19481
> URL: https://issues.apache.org/jira/browse/FLINK-19481
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
> Affects Versions: 1.12.0
> Reporter: Ben Augarten
> Priority: Major
>
> Currently, GCS is supported but only by using the hadoop connector[1]
>
> The objective of this improvement is to add support for checkpointing to Google Cloud Storage with the Flink File System,
>
> This would allow the `gs://` scheme to be used for savepointing and checkpointing. Long term, it would be nice if we could use the GCS FileSystem as a source and sink in flink jobs as well.
>
> Long term, I hope that implementing a flink native GCS FileSystem will simplify usage of GCS because the hadoop FileSystem ends up bringing in many unshaded dependencies.
>
> [1] [https://github.com/GoogleCloudDataproc/hadoop-connectors|https://github.com/GoogleCloudDataproc/hadoop-connectors)]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)