Using the KubernetesPodOperator

[Joe Napolitano <jo...@wework.com.INVALID>]

Hi All!

After almost an entire year, I finally hit a snag with Airflow that I can't
overcome :)

I'm currently trying to use the KubernetesPodOperator. Let's assume that I
have the RBAC and service account setup correctly as part of my Helm chart.

I'm creating the operator as follows, which closely follows the examples
online.

kubernetes_min_pod = KubernetesPodOperator(
    task_id='pod-ex-minimum',
    name='pod-ex-minimum',
    cmds=['echo'],
    namespace='dataplatform',
    service_account_name='my-service-account',
    in_cluster=True,
    image='gcr.io/gcp-runtimes/ubuntu_16_0_4',
    dag=dag)



What I'm seeing when I run the operator is the following exception:

HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
is forbidden: User \"system:serviceaccount:dataplatform:default\"
cannot create resource \"pods\" in API group \"\" in the namespace
\"dataplatform\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

Particulary, "User \"system:serviceaccount:dataplatform:default\" cannot
create"

Basically it keeps falling back to the default service account, instead of
using the one that I specified. This is why I believe this is isolated to
the operator code, and not related to my chart setup.

Unfortunately, I'm having trouble finding examples online that show usage
of the service account at all. And to make matters worse, I'm finding this
one difficult to troubleshoot because of the way the kubernetes library
works with the "in cluster" vs. "local" config files.

If anybody can clearly spot my mistake or give me a heads-up, it would be
greatly appreciated! Otherwise I will respond back with my result to create
a permanent record on the email chain.

Thanks, and all the best!
Joe Nap

Re: Using the KubernetesPodOperator

[Magnus Runesson <ma...@linuxalert.org>]

Hi Joe!

Your Airflow scheduler is not having the right to create pods. You must 
run the scheduler with an service account having the rights to create pods.

The service account you provide to "KubernetesPodOperator" is the 
service account your job/operator will run as.

/Magnus

On 2019-09-26 00:11, Joe Napolitano wrote:
> Hi All!
>
> After almost an entire year, I finally hit a snag with Airflow that I can't
> overcome :)
>
> I'm currently trying to use the KubernetesPodOperator. Let's assume that I
> have the RBAC and service account setup correctly as part of my Helm chart.
>
> I'm creating the operator as follows, which closely follows the examples
> online.
>
> kubernetes_min_pod = KubernetesPodOperator(
>      task_id='pod-ex-minimum',
>      name='pod-ex-minimum',
>      cmds=['echo'],
>      namespace='dataplatform',
>      service_account_name='my-service-account',
>      in_cluster=True,
>      image='gcr.io/gcp-runtimes/ubuntu_16_0_4',
>      dag=dag)
>
>
>
> What I'm seeing when I run the operator is the following exception:
>
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
> is forbidden: User \"system:serviceaccount:dataplatform:default\"
> cannot create resource \"pods\" in API group \"\" in the namespace
> \"dataplatform\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
>
> Particulary, "User \"system:serviceaccount:dataplatform:default\" cannot
> create"
>
> Basically it keeps falling back to the default service account, instead of
> using the one that I specified. This is why I believe this is isolated to
> the operator code, and not related to my chart setup.
>
> Unfortunately, I'm having trouble finding examples online that show usage
> of the service account at all. And to make matters worse, I'm finding this
> one difficult to troubleshoot because of the way the kubernetes library
> works with the "in cluster" vs. "local" config files.
>
> If anybody can clearly spot my mistake or give me a heads-up, it would be
> greatly appreciated! Otherwise I will respond back with my result to create
> a permanent record on the email chain.
>
> Thanks, and all the best!
> Joe Nap
>