Running Beam Pipelines on a Flink Application Mode Cluster

[Jan Bensien <st...@mail.uni-kiel.de>]

Hello,

I am currently trying to run my Apache Beam applications using Flink as 
my backend. Currently i use a session cluster running on Kubernetes. Is 
it possible to run Beam pipelines using the application mode? I would 
like to change to application mode, as I currently benchmark my 
applications with limited execution time and reset my cluster between 
every execution.

With many thanks,

Jan

Re: Running Beam Pipelines on a Flink Application Mode Cluster

[Yang Wang <da...@gmail.com>]

Hi Jan,

If you could run your Apache Beam application with Flink session mode, then
it could also
work in application mode. The key difference for application mode is that
the job submission happens
in the JobManager pod, not at the Flink client side. If you want to use the
standalone application
on k8s[1], then you need to clean up all the resources(e.g. JobManager Job,
TaskManager Deployment)
manually after every execution. Maybe you could also have a try on the
native Flink application on K8s[2],
all the resources will be cleaned automatically once the job reached the
terminal state.

Please note that you need to ensure your user jars already exist in the
JobManager pod before
starting the cluster entrypoint. Using the init container or baking the
user jars into the docker
image could both work.

[1].
https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/standalone/kubernetes.html#deploy-application-cluster
[2].
https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/native_kubernetes.html#application-mode


Best,
Yang

Jan Bensien <st...@mail.uni-kiel.de> 于2021年2月2日周二 上午5:04写道：

> Hello,
>
> I am currently trying to run my Apache Beam applications using Flink as my
> backend. Currently i use a session cluster running on Kubernetes. Is it
> possible to run Beam pipelines using the application mode? I would like to
> change to application mode, as I currently benchmark my applications with
> limited execution time and reset my cluster between every execution.
>
>
> With many thanks,
>
> Jan
>