You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Red Daly <re...@gmail.com> on 2022/05/21 06:00:47 UTC
Re: Example of Beam Go job that uses Flink Kubernetes operator
I created an example project with my attempts to get something working:
https://github.com/gonzojive/beam-go-k8s
The good news is I got something working with Beam, Go, Bazel, Flink, and
Kubernetes (minikube). The working version doesn't use the operator and
only runs in a single pod.
I would still like to figure out how to use the Flink operator with a Go
Beam job. I'm not yet sure how the operator should work together with the
"beam_flink1.14_job_server" and the Go binary.
On Fri, May 20, 2022 at 5:17 PM Red Daly <re...@gmail.com> wrote:
> Hi,
>
> This is a request for documentation to walk through using the Flink
> Kubernetes operator with the Go Beam SDK. I will plan to update the thread
> as I investigate. I hope this feedback is useful.
>
> A few observations:
>
> 1) The official "Quick Start guide
> <https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/>"
> for the Kubernetes operator is great. It would be great if there were a
> followup guide for how to use this operator to run a Beam job, especially a
> Beam Go job.
>
> 2) The closest to a walkthrough I have found is
> github.com/sambvfx/beam-flink-k8s, but that doesn't seem to use the
> operator at all.
>
> 3) There are 3 different Flink Kubernetes operators. (Lyft,
> GoogleCloudPlatform, and the Apache one.) The github repos for these don't
> have information about the other projects in their READMEs, from what I can
> tell. So it is a bit confusing.
>
> 4) There are a lot of concepts between Beam and Flink. To be a bit blunt,
> I don't want to learn many concepts up front. I just want to run a few
> commands and be up and running, then invest the time to learn more concepts
> later.
>
> - Red
>
Re: Example of Beam Go job that uses Flink Kubernetes operator
Posted by Gyula Fóra <gy...@gmail.com>.
Hi Red!
Thank you for your interest and for checking out the official Flink
Kubernetes Operator Project.
This project is still relatively new and so far our focus has been on
running native Flink workloads (Flink Application/Session clusters, and
session jobs) and we haven't tried running a Flink Beam job yet.
I think in your case the best thing to try would be the application
deployment
<https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/overview/#application-deployments>.
The current assumption is that your Flink application workload is packaged
into a fatjar with a main class that can be executed by the Flink jobmaster
component. I am not sure to what extent this is possible with the current
Beam implementation, it would be good to get input from the Beam community.
I guess there are multiple ways to achieve this on a high level:
- Bundle everything into a fatjar (runner + user jar + configs)
- Have the runner and user jars separately in the docker image and
configure flink to add them to the classpath
Maybe someone already tried running Beam jobs using the Flink Application
mode in Flink, that person should be able to provide a working answer :)
Cheers,
Gyula
On Sat, May 21, 2022 at 8:04 AM Red Daly <re...@gmail.com> wrote:
> I created an example project with my attempts to get something working:
>
> https://github.com/gonzojive/beam-go-k8s
>
> The good news is I got something working with Beam, Go, Bazel, Flink, and
> Kubernetes (minikube). The working version doesn't use the operator and
> only runs in a single pod.
>
> I would still like to figure out how to use the Flink operator with a Go
> Beam job. I'm not yet sure how the operator should work together with the
> "beam_flink1.14_job_server" and the Go binary.
>
> On Fri, May 20, 2022 at 5:17 PM Red Daly <re...@gmail.com> wrote:
>
>> Hi,
>>
>> This is a request for documentation to walk through using the Flink
>> Kubernetes operator with the Go Beam SDK. I will plan to update the thread
>> as I investigate. I hope this feedback is useful.
>>
>> A few observations:
>>
>> 1) The official "Quick Start guide
>> <https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/>"
>> for the Kubernetes operator is great. It would be great if there were a
>> followup guide for how to use this operator to run a Beam job, especially a
>> Beam Go job.
>>
>> 2) The closest to a walkthrough I have found is
>> github.com/sambvfx/beam-flink-k8s, but that doesn't seem to use the
>> operator at all.
>>
>> 3) There are 3 different Flink Kubernetes operators. (Lyft,
>> GoogleCloudPlatform, and the Apache one.) The github repos for these don't
>> have information about the other projects in their READMEs, from what I can
>> tell. So it is a bit confusing.
>>
>> 4) There are a lot of concepts between Beam and Flink. To be a bit blunt,
>> I don't want to learn many concepts up front. I just want to run a few
>> commands and be up and running, then invest the time to learn more concepts
>> later.
>>
>> - Red
>>
>
Re: Example of Beam Go job that uses Flink Kubernetes operator
Posted by Gyula Fóra <gy...@gmail.com>.
Hi Red!
Thank you for your interest and for checking out the official Flink
Kubernetes Operator Project.
This project is still relatively new and so far our focus has been on
running native Flink workloads (Flink Application/Session clusters, and
session jobs) and we haven't tried running a Flink Beam job yet.
I think in your case the best thing to try would be the application
deployment
<https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/overview/#application-deployments>.
The current assumption is that your Flink application workload is packaged
into a fatjar with a main class that can be executed by the Flink jobmaster
component. I am not sure to what extent this is possible with the current
Beam implementation, it would be good to get input from the Beam community.
I guess there are multiple ways to achieve this on a high level:
- Bundle everything into a fatjar (runner + user jar + configs)
- Have the runner and user jars separately in the docker image and
configure flink to add them to the classpath
Maybe someone already tried running Beam jobs using the Flink Application
mode in Flink, that person should be able to provide a working answer :)
Cheers,
Gyula
On Sat, May 21, 2022 at 8:04 AM Red Daly <re...@gmail.com> wrote:
> I created an example project with my attempts to get something working:
>
> https://github.com/gonzojive/beam-go-k8s
>
> The good news is I got something working with Beam, Go, Bazel, Flink, and
> Kubernetes (minikube). The working version doesn't use the operator and
> only runs in a single pod.
>
> I would still like to figure out how to use the Flink operator with a Go
> Beam job. I'm not yet sure how the operator should work together with the
> "beam_flink1.14_job_server" and the Go binary.
>
> On Fri, May 20, 2022 at 5:17 PM Red Daly <re...@gmail.com> wrote:
>
>> Hi,
>>
>> This is a request for documentation to walk through using the Flink
>> Kubernetes operator with the Go Beam SDK. I will plan to update the thread
>> as I investigate. I hope this feedback is useful.
>>
>> A few observations:
>>
>> 1) The official "Quick Start guide
>> <https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/>"
>> for the Kubernetes operator is great. It would be great if there were a
>> followup guide for how to use this operator to run a Beam job, especially a
>> Beam Go job.
>>
>> 2) The closest to a walkthrough I have found is
>> github.com/sambvfx/beam-flink-k8s, but that doesn't seem to use the
>> operator at all.
>>
>> 3) There are 3 different Flink Kubernetes operators. (Lyft,
>> GoogleCloudPlatform, and the Apache one.) The github repos for these don't
>> have information about the other projects in their READMEs, from what I can
>> tell. So it is a bit confusing.
>>
>> 4) There are a lot of concepts between Beam and Flink. To be a bit blunt,
>> I don't want to learn many concepts up front. I just want to run a few
>> commands and be up and running, then invest the time to learn more concepts
>> later.
>>
>> - Red
>>
>