Need help in setting up flink 1.10

[Ravi Sankar Reddy Sangana <Ra...@radware.com>]

Hi Team,

We are using flink 1.7 currently in prod. We want to move to latest release and trying 1.10. We are using google storage for our checkpoints.

In these release we need Hadoop to the integrated from our side. So following this link<https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/hadoop.html> we copied the flink-shaded-hadoop-2-uber-2.4.1-10.0.jar to the lib folder. Now I an getting this error while submitting the job.

Caused by: java.io.IOException: No FileSystem for scheme: gs

Can you please explain or show me a doc or any blog to solve this error.


flink-conf.yml
state.backend: rocksdb
state.checkpoints.dir: gs://xyz/flink/pacifier/flink-checkpoints

Regards,
Ravi Sankar Reddy

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

Re: Need help in setting up flink 1.10

[David Anderson <da...@apache.org>]

Ravi,

Note that questions like this are better suited for the user mailing list.

According to [1], google cloud storage is supported under the gcs: url
scheme. Also, since Flink 1.10, most filesystems must be loaded as plugins,
rather than from the lib directory [2].

I don't have experience with GCS, but I hope this helps.

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/internals/filesystems.html#implementations
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/filesystems/#pluggable-file-systems

Regards,
David

On Tue, Sep 29, 2020 at 3:04 AM Ravi Sankar Reddy Sangana <Ra...@radware.com>
wrote:

>
> Hi Team,
>
> We are using flink 1.7 currently in prod. We want to move to latest
> release and trying 1.10. We are using google storage for our checkpoints.
>
> In these release we need Hadoop to the integrated from our side. So
> following this link<
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/hadoop.html>
> we copied the flink-shaded-hadoop-2-uber-2.4.1-10.0.jar to the lib folder.
> Now I an getting this error while submitting the job.
>
> Caused by: java.io.IOException: No FileSystem for scheme: gs
>
> Can you please explain or show me a doc or any blog to solve this error.
>
>
> flink-conf.yml
> state.backend: rocksdb
> state.checkpoints.dir: gs://xyz/flink/pacifier/flink-checkpoints
>
> Regards,
> Ravi Sankar Reddy
>
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>