You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Xiao Ma <xi...@geotab.com> on 2022/05/17 03:37:09 UTC
Question of Flink Operator Application Cluster Deployment
Hi Flink Community,
First of all, I would like to express my great thankfulness about the flink
operator on Kubernetes. It is a new door to help us deploy the Flink
application on top of the K8s.
Our team is currently doing the Application cluster deployment through the
operator. We have set up the service account as "flink-operator" and
"flink", with the roles and rolebindings. However, after the job yaml is
submitted to the api-server and the pod is created, the resources manager
cannot be created because this error log:
====
2022-05-17 02:37:22,293 WARN io.fabric8.kubernetes.client.Config
[] - Error reading service account token from:
[/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
2022-05-17 02:37:22,308 WARN io.fabric8.kubernetes.client.Config
[] - Error reading service account token from:
[/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
2022-05-17 02:37:25,699 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Connecting to ResourceManager
akka.tcp://flink@flink-application-job.bip
:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)
2022-05-17 02:37:26,094 WARN
io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener [] -
Exec Failure: HTTP 403, Status: 403 - pods is forbidden: User
"system:anonymous" cannot watch resource "pods" in API group "" in the
namespace "xxxxxxxxx"
====
It looks like the jobmanager pod cannot fetch the "flink" service account
token and cannot communicate with api-server, though I have created the
"flink" service account and set up "serviceAccount" config in the job
template.
====
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: flink-application-job
spec:
image: flink:1.15.0-scala_2.12-java11
flinkVersion: v1_15
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
jobmanager.rpc.address: flink-jobmanager
serviceAccount: flink
====
The below shows the volumeMounts in the pod. The service account is mounted
through the "bound service account token volume". Is it desirable?
====
Mounts:
/opt/flink/conf from flink-config-volume (rw)
/opt/flink/log from flink-logs (rw)
/opt/flink/pod-template from pod-template-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from
kube-api-access-f69zl (ro)
====
This issue has blocked our progress for several days so if there are any
possible thoughts, we really appreciate it!
Thank you very much and I'm looking forward to your reply.
Best,
*Xiao Ma*
*Geotab*
Software Developer, Data Engineering | B.Sc, M.Sc
Direct +1 (416) 836 - 3541
Toll-free +1 (877) 436 - 8221
Visit www.geotab.com
Twitter <https://twitter.com/geotab> | Facebook
<https://www.facebook.com/Geotab> | YouTube
<https://www.youtube.com/user/MyGeotab> | LinkedIn
<https://www.linkedin.com/company/geotab/>
Re: Question of Flink Operator Application Cluster Deployment
Posted by Xiao Ma <xi...@geotab.com>.
Hi Őrhidi,
Thank you for helping out. I didn't try it on other k8s clusters. Our team
is on the whole GKE environment. Is the psp the possible cause? I have
given the secret volume in the psp, but not working.
Best,
*Xiao Ma*
*Geotab*
Software Developer, Data Engineering | B.Sc, M.Sc
Direct +1 (416) 836 - 3541
Toll-free +1 (877) 436 - 8221
Visit www.geotab.com
Twitter <https://twitter.com/geotab> | Facebook
<https://www.facebook.com/Geotab> | YouTube
<https://www.youtube.com/user/MyGeotab> | LinkedIn
<https://www.linkedin.com/company/geotab/>
On Wed, May 18, 2022 at 12:46 PM Őrhidi Mátyás <ma...@gmail.com>
wrote:
> Hi I couldn't spot anything wrong with your files. Actually I managed to
> run it on my local minikube. I suspect some environment specific issue
> here. I don't have access to a GKE instance unfortunately.
>
> Have you tried running it on other k8s clusters?
>
> Best,
> Matyas
>
> On Tue, May 17, 2022 at 4:55 PM Xiao Ma <xi...@geotab.com> wrote:
>
>> Hi Őrhidi,
>>
>> Thank you very much for the help.
>>
>> The attached are flink-operator yaml files and the application job yaml
>> file.
>>
>> Best,
>> *Xiao Ma*
>> *Geotab*
>> Software Developer, Data Engineering | B.Sc, M.Sc
>> Direct +1 (416) 836 - 3541
>> Toll-free +1 (877) 436 - 8221
>> Visit www.geotab.com
>> Twitter <https://twitter.com/geotab> | Facebook
>> <https://www.facebook.com/Geotab> | YouTube
>> <https://www.youtube.com/user/MyGeotab> | LinkedIn
>> <https://www.linkedin.com/company/geotab/>
>>
>>
>> On Tue, May 17, 2022 at 12:22 AM Őrhidi Mátyás <ma...@gmail.com>
>> wrote:
>>
>>> You don't have to mount the service account explicitly, this should
>>> be auto-mounted for you. Please share your (redacted) yamls for the RBAC
>>> configs (
>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/#cluster-scoped-flink-operator-with-jobs-running-in-other-namespaces)
>>> and your deployment yaml, we could probably spot what's missing.
>>>
>>> Best,
>>> Matyas
>>>
>>> On Tue, May 17, 2022 at 5:37 AM Xiao Ma <xi...@geotab.com> wrote:
>>>
>>>> Hi Flink Community,
>>>>
>>>> First of all, I would like to express my great thankfulness about the
>>>> flink operator on Kubernetes. It is a new door to help us deploy the Flink
>>>> application on top of the K8s.
>>>>
>>>> Our team is currently doing the Application cluster deployment through
>>>> the operator. We have set up the service account as "flink-operator" and
>>>> "flink", with the roles and rolebindings. However, after the job yaml is
>>>> submitted to the api-server and the pod is created, the resources manager
>>>> cannot be created because this error log:
>>>> ====
>>>> 2022-05-17 02:37:22,293 WARN io.fabric8.kubernetes.client.Config
>>>> [] - Error reading service account token from:
>>>> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
>>>> 2022-05-17 02:37:22,308 WARN io.fabric8.kubernetes.client.Config
>>>> [] - Error reading service account token from:
>>>> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
>>>> 2022-05-17 02:37:25,699 INFO
>>>> org.apache.flink.runtime.jobmaster.JobMaster [] -
>>>> Connecting to ResourceManager akka.tcp://flink@flink-application-job.bip
>>>> :6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)
>>>> 2022-05-17 02:37:26,094 WARN
>>>> io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener [] -
>>>> Exec Failure: HTTP 403, Status: 403 - pods is forbidden: User
>>>> "system:anonymous" cannot watch resource "pods" in API group "" in the
>>>> namespace "xxxxxxxxx"
>>>> ====
>>>>
>>>> It looks like the jobmanager pod cannot fetch the "flink" service
>>>> account token and cannot communicate with api-server, though I have created
>>>> the "flink" service account and set up "serviceAccount" config in the job
>>>> template.
>>>> ====
>>>>
>>>> apiVersion: flink.apache.org/v1beta1
>>>> kind: FlinkDeployment
>>>> metadata:
>>>> name: flink-application-job
>>>> spec:
>>>> image: flink:1.15.0-scala_2.12-java11
>>>> flinkVersion: v1_15
>>>> flinkConfiguration:
>>>> taskmanager.numberOfTaskSlots: "2"
>>>> jobmanager.rpc.address: flink-jobmanager
>>>> serviceAccount: flink
>>>>
>>>> ====
>>>>
>>>> The below shows the volumeMounts in the pod. The service account is
>>>> mounted through the "bound service account token volume". Is it desirable?
>>>> ====
>>>> Mounts:
>>>> /opt/flink/conf from flink-config-volume (rw)
>>>> /opt/flink/log from flink-logs (rw)
>>>> /opt/flink/pod-template from pod-template-volume (rw)
>>>> /var/run/secrets/kubernetes.io/serviceaccount from
>>>> kube-api-access-f69zl (ro)
>>>> ====
>>>>
>>>> This issue has blocked our progress for several days so if there are
>>>> any possible thoughts, we really appreciate it!
>>>>
>>>> Thank you very much and I'm looking forward to your reply.
>>>>
>>>>
>>>> Best,
>>>> *Xiao Ma*
>>>> *Geotab*
>>>> Software Developer, Data Engineering | B.Sc, M.Sc
>>>> Direct +1 (416) 836 - 3541
>>>> Toll-free +1 (877) 436 - 8221
>>>> Visit www.geotab.com
>>>> Twitter <https://twitter.com/geotab> | Facebook
>>>> <https://www.facebook.com/Geotab> | YouTube
>>>> <https://www.youtube.com/user/MyGeotab> | LinkedIn
>>>> <https://www.linkedin.com/company/geotab/>
>>>>
>>>
Re: Question of Flink Operator Application Cluster Deployment
Posted by Xiao Ma <xi...@geotab.com>.
Hi John,
No such deployment or services in the K8S cluster. Same issue happens to
the flink native kubernetes deployment. We have the podsecuritypolicy
defined, but I have added flink service account into the psp.
*Xiao Ma*
*Geotab*
Software Developer, Data Engineering | B.Sc, M.Sc
Direct +1 (416) 836 - 3541
Toll-free +1 (877) 436 - 8221
Visit www.geotab.com
Twitter <https://twitter.com/geotab> | Facebook
<https://www.facebook.com/Geotab> | YouTube
<https://www.youtube.com/user/MyGeotab> | LinkedIn
<https://www.linkedin.com/company/geotab/>
On Tue, May 17, 2022 at 9:50 PM John Gerassimou <jo...@unity3d.com>
wrote:
> Hi Xiao,
>
> Is istio or something similar deployed to the K8S cluster?
>
> John
>
> On Tue, May 17, 2022 at 4:26 PM Xiao Ma <xi...@geotab.com> wrote:
>
>> loop in
>> *Xiao Ma*
>> *Geotab*
>> Software Developer, Data Engineering | B.Sc, M.Sc
>> Direct +1 (416) 836 - 3541
>> Toll-free +1 (877) 436 - 8221
>> Visit www.geotab.com
>> Twitter <https://twitter.com/geotab> | Facebook
>> <https://www.facebook.com/Geotab> | YouTube
>> <https://www.youtube.com/user/MyGeotab> | LinkedIn
>> <https://www.linkedin.com/company/geotab/>
>>
>>
>> ---------- Forwarded message ---------
>> From: Xiao Ma <xi...@geotab.com>
>> Date: Tue, May 17, 2022 at 4:18 PM
>> Subject: Re: Question of Flink Operator Application Cluster Deployment
>> To: Őrhidi Mátyás <ma...@gmail.com>
>>
>>
>> Fyi, I didn't manually mount the service account token into the job pod.
>> It is automatically mounted into the pod, with the "bound service account
>> token volume". I also found that the fabric8 cannot read the service
>> account token if it is the "bound service account token volume". link:
>> https://github.com/fabric8io/kubernetes-client/issues/2271
>>
>> Thank you very much.
>>
>> Best,
>> *Xiao Ma*
>> *Geotab*
>> Software Developer, Data Engineering | B.Sc, M.Sc
>> Direct +1 (416) 836 - 3541
>> Toll-free +1 (877) 436 - 8221
>> Visit www.geotab.com
>> Twitter <https://twitter.com/geotab> | Facebook
>> <https://www.facebook.com/Geotab> | YouTube
>> <https://www.youtube.com/user/MyGeotab> | LinkedIn
>> <https://www.linkedin.com/company/geotab/>
>>
>>
>> On Tue, May 17, 2022 at 10:55 AM Xiao Ma <xi...@geotab.com> wrote:
>>
>>> Hi Őrhidi,
>>>
>>> Thank you very much for the help.
>>>
>>> The attached are flink-operator yaml files and the application job yaml
>>> file.
>>>
>>> Best,
>>> *Xiao Ma*
>>> *Geotab*
>>> Software Developer, Data Engineering | B.Sc, M.Sc
>>> Direct +1 (416) 836 - 3541
>>> Toll-free +1 (877) 436 - 8221
>>> Visit www.geotab.com
>>> Twitter <https://twitter.com/geotab> | Facebook
>>> <https://www.facebook.com/Geotab> | YouTube
>>> <https://www.youtube.com/user/MyGeotab> | LinkedIn
>>> <https://www.linkedin.com/company/geotab/>
>>>
>>>
>>> On Tue, May 17, 2022 at 12:22 AM Őrhidi Mátyás <ma...@gmail.com>
>>> wrote:
>>>
>>>> You don't have to mount the service account explicitly, this should
>>>> be auto-mounted for you. Please share your (redacted) yamls for the RBAC
>>>> configs (
>>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/#cluster-scoped-flink-operator-with-jobs-running-in-other-namespaces)
>>>> and your deployment yaml, we could probably spot what's missing.
>>>>
>>>> Best,
>>>> Matyas
>>>>
>>>> On Tue, May 17, 2022 at 5:37 AM Xiao Ma <xi...@geotab.com> wrote:
>>>>
>>>>> Hi Flink Community,
>>>>>
>>>>> First of all, I would like to express my great thankfulness about the
>>>>> flink operator on Kubernetes. It is a new door to help us deploy the Flink
>>>>> application on top of the K8s.
>>>>>
>>>>> Our team is currently doing the Application cluster deployment through
>>>>> the operator. We have set up the service account as "flink-operator" and
>>>>> "flink", with the roles and rolebindings. However, after the job yaml is
>>>>> submitted to the api-server and the pod is created, the resources manager
>>>>> cannot be created because this error log:
>>>>> ====
>>>>> 2022-05-17 02:37:22,293 WARN io.fabric8.kubernetes.client.Config
>>>>> [] - Error reading service account token from:
>>>>> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
>>>>> 2022-05-17 02:37:22,308 WARN io.fabric8.kubernetes.client.Config
>>>>> [] - Error reading service account token from:
>>>>> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
>>>>> 2022-05-17 02:37:25,699 INFO
>>>>> org.apache.flink.runtime.jobmaster.JobMaster [] -
>>>>> Connecting to ResourceManager akka.tcp://flink@flink-application-job.bip
>>>>> :6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)
>>>>> 2022-05-17 02:37:26,094 WARN
>>>>> io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener [] -
>>>>> Exec Failure: HTTP 403, Status: 403 - pods is forbidden: User
>>>>> "system:anonymous" cannot watch resource "pods" in API group "" in the
>>>>> namespace "xxxxxxxxx"
>>>>> ====
>>>>>
>>>>> It looks like the jobmanager pod cannot fetch the "flink" service
>>>>> account token and cannot communicate with api-server, though I have created
>>>>> the "flink" service account and set up "serviceAccount" config in the job
>>>>> template.
>>>>> ====
>>>>>
>>>>> apiVersion: flink.apache.org/v1beta1
>>>>> kind: FlinkDeployment
>>>>> metadata:
>>>>> name: flink-application-job
>>>>> spec:
>>>>> image: flink:1.15.0-scala_2.12-java11
>>>>> flinkVersion: v1_15
>>>>> flinkConfiguration:
>>>>> taskmanager.numberOfTaskSlots: "2"
>>>>> jobmanager.rpc.address: flink-jobmanager
>>>>> serviceAccount: flink
>>>>>
>>>>> ====
>>>>>
>>>>> The below shows the volumeMounts in the pod. The service account is
>>>>> mounted through the "bound service account token volume". Is it desirable?
>>>>> ====
>>>>> Mounts:
>>>>> /opt/flink/conf from flink-config-volume (rw)
>>>>> /opt/flink/log from flink-logs (rw)
>>>>> /opt/flink/pod-template from pod-template-volume (rw)
>>>>> /var/run/secrets/kubernetes.io/serviceaccount from
>>>>> kube-api-access-f69zl (ro)
>>>>> ====
>>>>>
>>>>> This issue has blocked our progress for several days so if there are
>>>>> any possible thoughts, we really appreciate it!
>>>>>
>>>>> Thank you very much and I'm looking forward to your reply.
>>>>>
>>>>>
>>>>> Best,
>>>>> *Xiao Ma*
>>>>> *Geotab*
>>>>> Software Developer, Data Engineering | B.Sc, M.Sc
>>>>> Direct +1 (416) 836 - 3541
>>>>> Toll-free +1 (877) 436 - 8221
>>>>> Visit www.geotab.com
>>>>> Twitter <https://twitter.com/geotab> | Facebook
>>>>> <https://www.facebook.com/Geotab> | YouTube
>>>>> <https://www.youtube.com/user/MyGeotab> | LinkedIn
>>>>> <https://www.linkedin.com/company/geotab/>
>>>>>
>>>>
Re: Question of Flink Operator Application Cluster Deployment
Posted by John Gerassimou <jo...@unity3d.com>.
Hi Xiao,
Is istio or something similar deployed to the K8S cluster?
John
On Tue, May 17, 2022 at 4:26 PM Xiao Ma <xi...@geotab.com> wrote:
> loop in
> *Xiao Ma*
> *Geotab*
> Software Developer, Data Engineering | B.Sc, M.Sc
> Direct +1 (416) 836 - 3541
> Toll-free +1 (877) 436 - 8221
> Visit www.geotab.com
> Twitter <https://twitter.com/geotab> | Facebook
> <https://www.facebook.com/Geotab> | YouTube
> <https://www.youtube.com/user/MyGeotab> | LinkedIn
> <https://www.linkedin.com/company/geotab/>
>
>
> ---------- Forwarded message ---------
> From: Xiao Ma <xi...@geotab.com>
> Date: Tue, May 17, 2022 at 4:18 PM
> Subject: Re: Question of Flink Operator Application Cluster Deployment
> To: Őrhidi Mátyás <ma...@gmail.com>
>
>
> Fyi, I didn't manually mount the service account token into the job pod.
> It is automatically mounted into the pod, with the "bound service account
> token volume". I also found that the fabric8 cannot read the service
> account token if it is the "bound service account token volume". link:
> https://github.com/fabric8io/kubernetes-client/issues/2271
>
> Thank you very much.
>
> Best,
> *Xiao Ma*
> *Geotab*
> Software Developer, Data Engineering | B.Sc, M.Sc
> Direct +1 (416) 836 - 3541
> Toll-free +1 (877) 436 - 8221
> Visit www.geotab.com
> Twitter <https://twitter.com/geotab> | Facebook
> <https://www.facebook.com/Geotab> | YouTube
> <https://www.youtube.com/user/MyGeotab> | LinkedIn
> <https://www.linkedin.com/company/geotab/>
>
>
> On Tue, May 17, 2022 at 10:55 AM Xiao Ma <xi...@geotab.com> wrote:
>
>> Hi Őrhidi,
>>
>> Thank you very much for the help.
>>
>> The attached are flink-operator yaml files and the application job yaml
>> file.
>>
>> Best,
>> *Xiao Ma*
>> *Geotab*
>> Software Developer, Data Engineering | B.Sc, M.Sc
>> Direct +1 (416) 836 - 3541
>> Toll-free +1 (877) 436 - 8221
>> Visit www.geotab.com
>> Twitter <https://twitter.com/geotab> | Facebook
>> <https://www.facebook.com/Geotab> | YouTube
>> <https://www.youtube.com/user/MyGeotab> | LinkedIn
>> <https://www.linkedin.com/company/geotab/>
>>
>>
>> On Tue, May 17, 2022 at 12:22 AM Őrhidi Mátyás <ma...@gmail.com>
>> wrote:
>>
>>> You don't have to mount the service account explicitly, this should
>>> be auto-mounted for you. Please share your (redacted) yamls for the RBAC
>>> configs (
>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/#cluster-scoped-flink-operator-with-jobs-running-in-other-namespaces)
>>> and your deployment yaml, we could probably spot what's missing.
>>>
>>> Best,
>>> Matyas
>>>
>>> On Tue, May 17, 2022 at 5:37 AM Xiao Ma <xi...@geotab.com> wrote:
>>>
>>>> Hi Flink Community,
>>>>
>>>> First of all, I would like to express my great thankfulness about the
>>>> flink operator on Kubernetes. It is a new door to help us deploy the Flink
>>>> application on top of the K8s.
>>>>
>>>> Our team is currently doing the Application cluster deployment through
>>>> the operator. We have set up the service account as "flink-operator" and
>>>> "flink", with the roles and rolebindings. However, after the job yaml is
>>>> submitted to the api-server and the pod is created, the resources manager
>>>> cannot be created because this error log:
>>>> ====
>>>> 2022-05-17 02:37:22,293 WARN io.fabric8.kubernetes.client.Config
>>>> [] - Error reading service account token from:
>>>> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
>>>> 2022-05-17 02:37:22,308 WARN io.fabric8.kubernetes.client.Config
>>>> [] - Error reading service account token from:
>>>> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
>>>> 2022-05-17 02:37:25,699 INFO
>>>> org.apache.flink.runtime.jobmaster.JobMaster [] -
>>>> Connecting to ResourceManager akka.tcp://flink@flink-application-job.bip
>>>> :6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)
>>>> 2022-05-17 02:37:26,094 WARN
>>>> io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener [] -
>>>> Exec Failure: HTTP 403, Status: 403 - pods is forbidden: User
>>>> "system:anonymous" cannot watch resource "pods" in API group "" in the
>>>> namespace "xxxxxxxxx"
>>>> ====
>>>>
>>>> It looks like the jobmanager pod cannot fetch the "flink" service
>>>> account token and cannot communicate with api-server, though I have created
>>>> the "flink" service account and set up "serviceAccount" config in the job
>>>> template.
>>>> ====
>>>>
>>>> apiVersion: flink.apache.org/v1beta1
>>>> kind: FlinkDeployment
>>>> metadata:
>>>> name: flink-application-job
>>>> spec:
>>>> image: flink:1.15.0-scala_2.12-java11
>>>> flinkVersion: v1_15
>>>> flinkConfiguration:
>>>> taskmanager.numberOfTaskSlots: "2"
>>>> jobmanager.rpc.address: flink-jobmanager
>>>> serviceAccount: flink
>>>>
>>>> ====
>>>>
>>>> The below shows the volumeMounts in the pod. The service account is
>>>> mounted through the "bound service account token volume". Is it desirable?
>>>> ====
>>>> Mounts:
>>>> /opt/flink/conf from flink-config-volume (rw)
>>>> /opt/flink/log from flink-logs (rw)
>>>> /opt/flink/pod-template from pod-template-volume (rw)
>>>> /var/run/secrets/kubernetes.io/serviceaccount from
>>>> kube-api-access-f69zl (ro)
>>>> ====
>>>>
>>>> This issue has blocked our progress for several days so if there are
>>>> any possible thoughts, we really appreciate it!
>>>>
>>>> Thank you very much and I'm looking forward to your reply.
>>>>
>>>>
>>>> Best,
>>>> *Xiao Ma*
>>>> *Geotab*
>>>> Software Developer, Data Engineering | B.Sc, M.Sc
>>>> Direct +1 (416) 836 - 3541
>>>> Toll-free +1 (877) 436 - 8221
>>>> Visit www.geotab.com
>>>> Twitter <https://twitter.com/geotab> | Facebook
>>>> <https://www.facebook.com/Geotab> | YouTube
>>>> <https://www.youtube.com/user/MyGeotab> | LinkedIn
>>>> <https://www.linkedin.com/company/geotab/>
>>>>
>>>
Fwd: Question of Flink Operator Application Cluster Deployment
Posted by Xiao Ma <xi...@geotab.com>.
loop in
*Xiao Ma*
*Geotab*
Software Developer, Data Engineering | B.Sc, M.Sc
Direct +1 (416) 836 - 3541
Toll-free +1 (877) 436 - 8221
Visit www.geotab.com
Twitter <https://twitter.com/geotab> | Facebook
<https://www.facebook.com/Geotab> | YouTube
<https://www.youtube.com/user/MyGeotab> | LinkedIn
<https://www.linkedin.com/company/geotab/>
---------- Forwarded message ---------
From: Xiao Ma <xi...@geotab.com>
Date: Tue, May 17, 2022 at 4:18 PM
Subject: Re: Question of Flink Operator Application Cluster Deployment
To: Őrhidi Mátyás <ma...@gmail.com>
Fyi, I didn't manually mount the service account token into the job pod. It
is automatically mounted into the pod, with the "bound service account
token volume". I also found that the fabric8 cannot read the service
account token if it is the "bound service account token volume". link:
https://github.com/fabric8io/kubernetes-client/issues/2271
Thank you very much.
Best,
*Xiao Ma*
*Geotab*
Software Developer, Data Engineering | B.Sc, M.Sc
Direct +1 (416) 836 - 3541
Toll-free +1 (877) 436 - 8221
Visit www.geotab.com
Twitter <https://twitter.com/geotab> | Facebook
<https://www.facebook.com/Geotab> | YouTube
<https://www.youtube.com/user/MyGeotab> | LinkedIn
<https://www.linkedin.com/company/geotab/>
On Tue, May 17, 2022 at 10:55 AM Xiao Ma <xi...@geotab.com> wrote:
> Hi Őrhidi,
>
> Thank you very much for the help.
>
> The attached are flink-operator yaml files and the application job yaml
> file.
>
> Best,
> *Xiao Ma*
> *Geotab*
> Software Developer, Data Engineering | B.Sc, M.Sc
> Direct +1 (416) 836 - 3541
> Toll-free +1 (877) 436 - 8221
> Visit www.geotab.com
> Twitter <https://twitter.com/geotab> | Facebook
> <https://www.facebook.com/Geotab> | YouTube
> <https://www.youtube.com/user/MyGeotab> | LinkedIn
> <https://www.linkedin.com/company/geotab/>
>
>
> On Tue, May 17, 2022 at 12:22 AM Őrhidi Mátyás <ma...@gmail.com>
> wrote:
>
>> You don't have to mount the service account explicitly, this should
>> be auto-mounted for you. Please share your (redacted) yamls for the RBAC
>> configs (
>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/#cluster-scoped-flink-operator-with-jobs-running-in-other-namespaces)
>> and your deployment yaml, we could probably spot what's missing.
>>
>> Best,
>> Matyas
>>
>> On Tue, May 17, 2022 at 5:37 AM Xiao Ma <xi...@geotab.com> wrote:
>>
>>> Hi Flink Community,
>>>
>>> First of all, I would like to express my great thankfulness about the
>>> flink operator on Kubernetes. It is a new door to help us deploy the Flink
>>> application on top of the K8s.
>>>
>>> Our team is currently doing the Application cluster deployment through
>>> the operator. We have set up the service account as "flink-operator" and
>>> "flink", with the roles and rolebindings. However, after the job yaml is
>>> submitted to the api-server and the pod is created, the resources manager
>>> cannot be created because this error log:
>>> ====
>>> 2022-05-17 02:37:22,293 WARN io.fabric8.kubernetes.client.Config
>>> [] - Error reading service account token from:
>>> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
>>> 2022-05-17 02:37:22,308 WARN io.fabric8.kubernetes.client.Config
>>> [] - Error reading service account token from:
>>> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
>>> 2022-05-17 02:37:25,699 INFO
>>> org.apache.flink.runtime.jobmaster.JobMaster [] -
>>> Connecting to ResourceManager akka.tcp://flink@flink-application-job.bip
>>> :6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)
>>> 2022-05-17 02:37:26,094 WARN
>>> io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener [] -
>>> Exec Failure: HTTP 403, Status: 403 - pods is forbidden: User
>>> "system:anonymous" cannot watch resource "pods" in API group "" in the
>>> namespace "xxxxxxxxx"
>>> ====
>>>
>>> It looks like the jobmanager pod cannot fetch the "flink" service
>>> account token and cannot communicate with api-server, though I have created
>>> the "flink" service account and set up "serviceAccount" config in the job
>>> template.
>>> ====
>>>
>>> apiVersion: flink.apache.org/v1beta1
>>> kind: FlinkDeployment
>>> metadata:
>>> name: flink-application-job
>>> spec:
>>> image: flink:1.15.0-scala_2.12-java11
>>> flinkVersion: v1_15
>>> flinkConfiguration:
>>> taskmanager.numberOfTaskSlots: "2"
>>> jobmanager.rpc.address: flink-jobmanager
>>> serviceAccount: flink
>>>
>>> ====
>>>
>>> The below shows the volumeMounts in the pod. The service account is
>>> mounted through the "bound service account token volume". Is it desirable?
>>> ====
>>> Mounts:
>>> /opt/flink/conf from flink-config-volume (rw)
>>> /opt/flink/log from flink-logs (rw)
>>> /opt/flink/pod-template from pod-template-volume (rw)
>>> /var/run/secrets/kubernetes.io/serviceaccount from
>>> kube-api-access-f69zl (ro)
>>> ====
>>>
>>> This issue has blocked our progress for several days so if there are any
>>> possible thoughts, we really appreciate it!
>>>
>>> Thank you very much and I'm looking forward to your reply.
>>>
>>>
>>> Best,
>>> *Xiao Ma*
>>> *Geotab*
>>> Software Developer, Data Engineering | B.Sc, M.Sc
>>> Direct +1 (416) 836 - 3541
>>> Toll-free +1 (877) 436 - 8221
>>> Visit www.geotab.com
>>> Twitter <https://twitter.com/geotab> | Facebook
>>> <https://www.facebook.com/Geotab> | YouTube
>>> <https://www.youtube.com/user/MyGeotab> | LinkedIn
>>> <https://www.linkedin.com/company/geotab/>
>>>
>>
Re: Question of Flink Operator Application Cluster Deployment
Posted by Őrhidi Mátyás <ma...@gmail.com>.
You don't have to mount the service account explicitly, this should
be auto-mounted for you. Please share your (redacted) yamls for the RBAC
configs (
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/#cluster-scoped-flink-operator-with-jobs-running-in-other-namespaces)
and your deployment yaml, we could probably spot what's missing.
Best,
Matyas
On Tue, May 17, 2022 at 5:37 AM Xiao Ma <xi...@geotab.com> wrote:
> Hi Flink Community,
>
> First of all, I would like to express my great thankfulness about the
> flink operator on Kubernetes. It is a new door to help us deploy the Flink
> application on top of the K8s.
>
> Our team is currently doing the Application cluster deployment through the
> operator. We have set up the service account as "flink-operator" and
> "flink", with the roles and rolebindings. However, after the job yaml is
> submitted to the api-server and the pod is created, the resources manager
> cannot be created because this error log:
> ====
> 2022-05-17 02:37:22,293 WARN io.fabric8.kubernetes.client.Config
> [] - Error reading service account token from:
> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
> 2022-05-17 02:37:22,308 WARN io.fabric8.kubernetes.client.Config
> [] - Error reading service account token from:
> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
> 2022-05-17 02:37:25,699 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Connecting to ResourceManager
> akka.tcp://flink@flink-application-job.bip
> :6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)
> 2022-05-17 02:37:26,094 WARN
> io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener [] -
> Exec Failure: HTTP 403, Status: 403 - pods is forbidden: User
> "system:anonymous" cannot watch resource "pods" in API group "" in the
> namespace "xxxxxxxxx"
> ====
>
> It looks like the jobmanager pod cannot fetch the "flink" service account
> token and cannot communicate with api-server, though I have created the
> "flink" service account and set up "serviceAccount" config in the job
> template.
> ====
>
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkDeployment
> metadata:
> name: flink-application-job
> spec:
> image: flink:1.15.0-scala_2.12-java11
> flinkVersion: v1_15
> flinkConfiguration:
> taskmanager.numberOfTaskSlots: "2"
> jobmanager.rpc.address: flink-jobmanager
> serviceAccount: flink
>
> ====
>
> The below shows the volumeMounts in the pod. The service account is
> mounted through the "bound service account token volume". Is it desirable?
> ====
> Mounts:
> /opt/flink/conf from flink-config-volume (rw)
> /opt/flink/log from flink-logs (rw)
> /opt/flink/pod-template from pod-template-volume (rw)
> /var/run/secrets/kubernetes.io/serviceaccount from
> kube-api-access-f69zl (ro)
> ====
>
> This issue has blocked our progress for several days so if there are any
> possible thoughts, we really appreciate it!
>
> Thank you very much and I'm looking forward to your reply.
>
>
> Best,
> *Xiao Ma*
> *Geotab*
> Software Developer, Data Engineering | B.Sc, M.Sc
> Direct +1 (416) 836 - 3541
> Toll-free +1 (877) 436 - 8221
> Visit www.geotab.com
> Twitter <https://twitter.com/geotab> | Facebook
> <https://www.facebook.com/Geotab> | YouTube
> <https://www.youtube.com/user/MyGeotab> | LinkedIn
> <https://www.linkedin.com/company/geotab/>
>