You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by P Singh <bi...@gmail.com> on 2023/02/03 12:32:07 UTC

How to submit beam python pipeline to GKE flink cluster

Hi Team,

I have set up a flink cluster on GKE and am trying to submit a beam
pipeline with below options. I was able to run this on a local machine but
I don't understand what would be the environment_config? What should I do?
what to put here instead of localhost:50000

Please help.
options = PipelineOptions([
"--runner=FlinkRunner",
"--flink_version=1.14",
"--flink_master=localhost:8081",
"--environment_type=EXTERNAL", #EXTERNAL
"--environment_config=localhost:50000",
])

Re: How to submit beam python pipeline to GKE flink cluster

Posted by P Singh <bi...@gmail.com>.
Hi,

I have done this manually. though It seems the beam version issues, I am
using 2.43. Is it compatible with flink 1.14 and python 3.10. Most of the
threads indicating beam doesn't really support latest versions.

On Sat, 4 Feb 2023 at 02:51, Robert Bradshaw via user <us...@beam.apache.org>
wrote:

> You should be able to omit the environment_type and environment_config
> variables and they will be populated automatically. For running
> locally, the flink_master parameter is not needed either (one will be
> started up automatically).
>
> On Fri, Feb 3, 2023 at 12:51 PM Talat Uyarer via user
> <us...@beam.apache.org> wrote:
> >
> > Hi,
> >
> > Do you use Flink operator or manually deployed session cluster ?
> >
> > Thanks
> >
> > On Fri, Feb 3, 2023, 4:32 AM P Singh <bi...@gmail.com>
> wrote:
> >>
> >> Hi Team,
> >>
> >> I have set up a flink cluster on GKE and am trying to submit a beam
> pipeline with below options. I was able to run this on a local machine but
> I don't understand what would be the environment_config? What should I do?
> what to put here instead of localhost:50000
> >>
> >> Please help.
> >> options = PipelineOptions([
> >> "--runner=FlinkRunner",
> >> "--flink_version=1.14",
> >> "--flink_master=localhost:8081",
> >> "--environment_type=EXTERNAL", #EXTERNAL
> >> "--environment_config=localhost:50000",
> >> ])
>

Re: How to submit beam python pipeline to GKE flink cluster

Posted by Robert Bradshaw via user <us...@beam.apache.org>.
You should be able to omit the environment_type and environment_config
variables and they will be populated automatically. For running
locally, the flink_master parameter is not needed either (one will be
started up automatically).

On Fri, Feb 3, 2023 at 12:51 PM Talat Uyarer via user
<us...@beam.apache.org> wrote:
>
> Hi,
>
> Do you use Flink operator or manually deployed session cluster ?
>
> Thanks
>
> On Fri, Feb 3, 2023, 4:32 AM P Singh <bi...@gmail.com> wrote:
>>
>> Hi Team,
>>
>> I have set up a flink cluster on GKE and am trying to submit a beam pipeline with below options. I was able to run this on a local machine but I don't understand what would be the environment_config? What should I do? what to put here instead of localhost:50000
>>
>> Please help.
>> options = PipelineOptions([
>> "--runner=FlinkRunner",
>> "--flink_version=1.14",
>> "--flink_master=localhost:8081",
>> "--environment_type=EXTERNAL", #EXTERNAL
>> "--environment_config=localhost:50000",
>> ])

Re: How to submit beam python pipeline to GKE flink cluster

Posted by Talat Uyarer via user <us...@beam.apache.org>.
Hi,

Do you use Flink operator or manually deployed session cluster ?

Thanks

On Fri, Feb 3, 2023, 4:32 AM P Singh <bi...@gmail.com> wrote:

> Hi Team,
>
> I have set up a flink cluster on GKE and am trying to submit a beam
> pipeline with below options. I was able to run this on a local machine but
> I don't understand what would be the environment_config? What should I do?
> what to put here instead of localhost:50000
>
> Please help.
> options = PipelineOptions([
> "--runner=FlinkRunner",
> "--flink_version=1.14",
> "--flink_master=localhost:8081",
> "--environment_type=EXTERNAL", #EXTERNAL
> "--environment_config=localhost:50000",
> ])
>