You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Tamir Sagi <Ta...@niceactimize.com> on 2022/11/22 11:44:27 UTC
[Knowledge Transfer] - Flink Native kubernetes + Istio
Hey All,
We recently encountered an issue while deploying application clusters on EKS along with Istio. I though it would be worth sharing with the community.
Our Flink pods are attached with service account annotated with IAM role to access AWS services.
The image used for the clusters is a custom Docker image based on Flink official image with minor modification.(More jars under lib and plugin folders)
While the cluster is created, Istio injects a Sidecard to every pod. The problem is that a short time after the cluster is up and running , it fails with AccessDenied(403) error against several AWS. (It works fine without Istio).
There was a suspicious log which needed an attention.
[Warning] {} [i.f.k.c.Config]: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
In general, Istio injects SecurityContext with fsGroup 1337 to the pod itself (The screenshots below have been taken from the pod's yaml)
Before enabling Istio
[cid:202858c3-94d8-4ed6-bcf7-420f19bfaca4]
After enabling Istio
[cid:27c42512-2f1a-4119-b78a-b6cd962f9c91]
Flink pods are running under 'flink' user (can be verified by running 'whoami' command within the pod)
https://github.com/apache/flink-docker/blob/master/1.15/scala_2.12-java11-ubuntu/Dockerfile#L55-L56
Checking permissions on the path /var/run/secrets/kubernetes.io/serviceaccount/
Before enabling Istio
[cid:500d6a05-7da3-4c58-8ad8-e89c7e27f3d2]
After enabling, the group is 1337
[cid:2918eca5-59cd-48e4-a9ff-80660aa4cb66]
We tried several manipulations, what worked eventually was to create a group with id 1337 and add 'flink' user to that group.
Adding this to our custom Dockerfile
RUN groupadd --system --gid=1337 istio \
&& usermod -a -G flink,istio flink
then, the group has been changed to 'istio'
[cid:6ba62570-d899-4682-9f73-64f77a8efad4]
This time, the cluster was up and running properly.
Best,
Tamir.
Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Re: [Knowledge Transfer] - Flink Native kubernetes + Istio
Posted by Tamir Sagi <Ta...@niceactimize.com>.
Hey Yang,
Thanks for your feedback.
We use Pod template as well. Istio adds fsGroup:1337 to the security context anyway. Creating a group with the same id as in security context and add 'flink' user onto that group was necessary.
Best,
Tamir.
________________________________
From: Yang Wang <da...@gmail.com>
Sent: Thursday, November 24, 2022 4:06 PM
To: Tamir Sagi <Ta...@niceactimize.com>
Cc: user@flink.apache.org <us...@flink.apache.org>
Subject: Re: [Knowledge Transfer] - Flink Native kubernetes + Istio
EXTERNAL EMAIL
Hi Tamir,
Thanks for sharing such a useful resource. I believe it will help a lot for other users when deploying Flink on K8s with Istio.
Do you think it is feasible to configure the security context via pod template[1]? Or the security context will always be overridden by Istio even it is configured explicitly.
[1]. https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#pod-template
Best,
Yang
Tamir Sagi <Ta...@niceactimize.com>> 于2022年11月22日周二 19:45写道：
Hey All,
We recently encountered an issue while deploying application clusters on EKS along with Istio. I though it would be worth sharing with the community.
Our Flink pods are attached with service account annotated with IAM role to access AWS services.
The image used for the clusters is a custom Docker image based on Flink official image with minor modification.(More jars under lib and plugin folders)
While the cluster is created, Istio injects a Sidecard to every pod. The problem is that a short time after the cluster is up and running , it fails with AccessDenied(403) error against several AWS. (It works fine without Istio).
There was a suspicious log which needed an attention.
[Warning] {} [i.f.k.c.Config]: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token<http://kubernetes.io/serviceaccount/token>]. Ignoring.
In general, Istio injects SecurityContext with fsGroup 1337 to the pod itself (The screenshots below have been taken from the pod's yaml)
Before enabling Istio
[cid:184a9f21e4bcb971f161]
After enabling Istio
[cid:184a9f21e4ccb971f164]
Flink pods are running under 'flink' user (can be verified by running 'whoami' command within the pod)
https://github.com/apache/flink-docker/blob/master/1.15/scala_2.12-java11-ubuntu/Dockerfile#L55-L56
Checking permissions on the path /var/run/secrets/kubernetes.io/serviceaccount/<http://kubernetes.io/serviceaccount/>
Before enabling Istio
[cid:184a9f21e4ccb971f165]
After enabling, the group is 1337
[cid:184a9f21e4ccb971f162]
We tried several manipulations, what worked eventually was to create a group with id 1337 and add 'flink' user to that group.
Adding this to our custom Dockerfile
RUN groupadd --system --gid=1337 istio \
&& usermod -a -G flink,istio flink
then, the group has been changed to 'istio'
[cid:184a9f21e4ccb971f163]
This time, the cluster was up and running properly.
Best,
Tamir.
Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Re: [Knowledge Transfer] - Flink Native kubernetes + Istio
Posted by Yang Wang <da...@gmail.com>.
Hi Tamir,
Thanks for sharing such a useful resource. I believe it will help a lot for
other users when deploying Flink on K8s with Istio.
Do you think it is feasible to configure the security context via pod
template[1]? Or the security context will always be overridden by Istio
even it is configured explicitly.
[1].
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#pod-template
Best,
Yang
Tamir Sagi <Ta...@niceactimize.com> 于2022年11月22日周二 19:45写道：
> Hey All,
>
> We recently encountered an issue while deploying application clusters on
> EKS along with Istio. I though it would be worth sharing with the community.
>
> Our Flink pods are attached with service account annotated with IAM role
> to access AWS services.
> The image used for the clusters is a custom Docker image based on Flink
> official image with minor modification.(More jars under lib and plugin
> folders)
>
> While the cluster is created, Istio injects a Sidecard to every pod. The
> problem is that a short time after the cluster is up and running , it
> fails with AccessDenied(403) error against several AWS. (It works fine
> without Istio).
>
> There was a suspicious log which needed an attention.
> [Warning] {} [i.f.k.c.Config]: Error reading service account token from:
> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
>
> In general, Istio injects SecurityContext with fsGroup 1337 to the pod
> itself (The screenshots below have been taken from the pod's yaml)
>
> Before enabling Istio
>
>
> After enabling Istio
>
>
> Flink pods are running under 'flink' user (can be verified by running
> 'whoami' command within the pod)
>
> https://github.com/apache/flink-docker/blob/master/1.15/scala_2.12-java11-ubuntu/Dockerfile#L55-L56
>
> Checking permissions on the path /var/run/secrets/
> kubernetes.io/serviceaccount/
>
> Before enabling Istio
>
>
> After enabling, the group is 1337
>
>
> We tried several manipulations, what worked eventually was to create a
> group with id 1337 and add 'flink' user to that group.
>
> Adding this to our custom Dockerfile
>
> RUN groupadd --system --gid=1337 istio \
> && usermod -a -G flink,istio flink
>
> then, the group has been changed to 'istio'
>
>
> This time, the cluster was up and running properly.
>
> Best,
> Tamir.
>
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>