You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Javier Vegas <jv...@strava.com> on 2022/07/22 10:03:23 UTC
NodePort conflict for multiple HA application-mode standalone Kubernetes deploys in same namespace
I am deploying a high-availability Flink job to Kubernetes in application
mode using Flink's standalone k8 deployment
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/kubernetes/
All goes well when I deploy a job, but if I want to deploy a second
application-mode Flink job in the same K8s namespace I get a
"spec.ports[0].nodePort:
Invalid value: 30081: provided port is already allocated" error. Is there a
way that nodePort can be allocated dynamically, or other way around this
(using Loadbalancer or Ingress instead of NodePort in
jobmanager-rest-service.yaml?) besides hard-coding different nodePorts for
different jobs running in same namespace?
Thanks,
Javier Vegas
Re: NodePort conflict for multiple HA application-mode standalone Kubernetes deploys in same namespace
Posted by Yang Wang <da...@gmail.com>.
Removing the nodePort for every different Flink application is necessary so
that it could pick up a random port.
Moreover, I believe you also need to change some other yamls. For example,
having a different name for JobManager/TaskManager yamls, update
the jobmanager-service.yaml and flink-configuration-configmap.yaml to use
the new name.
An easy way is to use the flink-kubernetes-operator[1].
[1].
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.1/
Best,
Yang
Javier Vegas <jv...@strava.com> 于2022年7月24日周日 23:43写道：
> Partial answer to my own question: Removing the hardcoded `nodePort:
> 30081` entry from jobmanager-rest-service.yaml, Flink assigns random
> ports so there are no conflicts and multiple Flink application-mode jobs
> can be deployed. However the jobs seem to communicate with each other, when
> launching the second job, the first job taskmanagers start executing tasks
> sent by the second job jobmanager, and the second job taskmanagers execute
> jobs from both jobmanagers.
>
> El vie, 22 jul 2022 a las 12:03, Javier Vegas (<jv...@strava.com>)
> escribió:
>
>>
>> I am deploying a high-availability Flink job to Kubernetes in application
>> mode using Flink's standalone k8 deployment
>> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/kubernetes/
>> All goes well when I deploy a job, but if I want to deploy a second
>> application-mode Flink job in the same K8s namespace I get a "spec.ports[0].nodePort:
>> Invalid value: 30081: provided port is already allocated" error. Is
>> there a way that nodePort can be allocated dynamically, or other way around
>> this (using Loadbalancer or Ingress instead of NodePort in
>> jobmanager-rest-service.yaml?) besides hard-coding different nodePorts
>> for different jobs running in same namespace?
>>
>> Thanks,
>>
>> Javier Vegas
>>
>
Re: NodePort conflict for multiple HA application-mode standalone Kubernetes deploys in same namespace
Posted by Javier Vegas <jv...@strava.com>.
Partial answer to my own question: Removing the hardcoded `nodePort:
30081` entry
from jobmanager-rest-service.yaml, Flink assigns random ports so there are
no conflicts and multiple Flink application-mode jobs can be deployed.
However the jobs seem to communicate with each other, when launching the
second job, the first job taskmanagers start executing tasks sent by the
second job jobmanager, and the second job taskmanagers execute jobs from
both jobmanagers.
El vie, 22 jul 2022 a las 12:03, Javier Vegas (<jv...@strava.com>)
escribió:
>
> I am deploying a high-availability Flink job to Kubernetes in application
> mode using Flink's standalone k8 deployment
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/kubernetes/
> All goes well when I deploy a job, but if I want to deploy a second
> application-mode Flink job in the same K8s namespace I get a "spec.ports[0].nodePort:
> Invalid value: 30081: provided port is already allocated" error. Is there
> a way that nodePort can be allocated dynamically, or other way around this
> (using Loadbalancer or Ingress instead of NodePort in
> jobmanager-rest-service.yaml?) besides hard-coding different nodePorts
> for different jobs running in same namespace?
>
> Thanks,
>
> Javier Vegas
>