You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "benrifkind (via GitHub)" <gi...@apache.org> on 2026/05/20 20:56:23 UTC

[I] workers.celery.serviceAccount.name is not used by Celery worker templates or RBAC bindings [airflow]

benrifkind opened a new issue, #67261:
URL: https://github.com/apache/airflow/issues/67261

   ### Under which category would you file this issue?
   
   Helm chart
   
   ### Apache Airflow version
   
   3.2.1
   
   ### What happened and how to reproduce it?
   
   When I deployed using the helm chart 1.21.0 by setting `workers.celery.serviceAccount.name` it had no effect on the Celery worker deployment, service account creation, or RBAC role bindings. 
   
   ### What you think should happen instead?
   
   The release notes of the Helm release 1.21.0 say
   > workers.serviceAccount is now deprecated in favor of workers.celery.serviceAccount/workers.kubernetes.serviceAccount (https://github.com/apache/airflow/pull/64730).
   
   I thought that meant that I should set `worker.celery.ServiceAccount` but it seems to have no affect. Keeping the configuration at `worker.ServiceAccount` fixes the issue. I am not sure if I am misunderstanding the release note.
   
   ## Expected behavior
   
   Setting `workers.celery.serviceAccount.name = "my-custom-sa"` should result in:
   - The Celery worker deployment using that service account
   - The RBAC role bindings (pod-launcher, job-launcher) referencing that service account
   - The worker service account resource being created with that name (if `create = true`)
   
   ## Actual behavior
   
   All Celery worker templates use the `worker.serviceAccountName` helper, which reads from `workers.serviceAccount` (the deprecated path) and never checks `workers.celery.serviceAccount`. When a user only sets the new path, the helper falls back to generating a default name (`{{ .Release.Name }}-worker`), ignoring the user's configuration.
   
   ## Root cause
   
   PR #64730 added `workers.celery.serviceAccount` and `workers.kubernetes.serviceAccount` as new config sections and deprecated `workers.serviceAccount`. The PR correctly created a `worker.kubernetes.serviceAccountName` helper that reads from `workers.kubernetes.serviceAccount` via `subKey = "kubernetes"`. However, no equivalent `worker.celery.serviceAccountName` helper was created.
   
   There is no corresponding `worker.celery.serviceAccountName`.
   https://github.com/apache/airflow/blob/helm-chart/1.21.0/chart/templates/_helpers.yaml#L724-L736
   
   
   ### Operating System
   
   PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Apache Airflow Provider(s)
   
   _No response_
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Official Helm Chart version
   
   1.21.0 (latest released)
   
   ### Kubernetes Version
   
   Server is v1.33.10 (EKS), client is v1.30.2.
   
   ### Helm Chart configuration
   
   _No response_
   
   ### Docker Image customizations
   
   Not applicable
   
   ### Anything else?
   
   Every time
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] workers.celery.serviceAccount.name is not used by Celery worker templates or RBAC bindings [airflow]

Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 closed issue #67261: workers.celery.serviceAccount.name is not used by Celery worker templates or RBAC bindings
URL: https://github.com/apache/airflow/issues/67261


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] workers.celery.serviceAccount.name is not used by Celery worker templates or RBAC bindings [airflow]

Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on issue #67261:
URL: https://github.com/apache/airflow/issues/67261#issuecomment-4502896554

   @benrifkind could you please add the Helm configuration? It can be hard to find an issue without it, as it is connected to Helm.
   
   > The RBAC role bindings (pod-launcher, job-launcher) referencing that service account
   
   It should, and in general it does, but not always, as I checked. I will create a fix (if someone is not quicker than me) in the upcoming days. I'm not sure if that will make it to the 1.22 release, as we are already two days past the cutoff. @potiuk wdyt as the current release manager for that release?
   
   ---
   
   For the rest points from expected behaviour, so:
   > The Celery worker deployment using that service account
   
   It should work correctly (we have tests for it), as I checked locally:
   *command:*
   ```shell
   helm template test chart/ --show-only templates/workers/worker-deployment.yaml --set workers.celery.serviceAccount.create=false,workers.celery.serviceAccount.name=test-test-test | grep serv
   iceAccount
   ```
   *results in:*
   ```yaml
   serviceAccountName: "test-test-test"
   ```
   Also, the `workers-serviceaccount.yaml` template is not rendered here, so as expected.
   
   > The worker service account resource being created with that name (if create = true)
   
   Same as above:
   *command:*
   ```shell
   helm template test chart/ --show-only templates/workers/worker-serviceaccount.yaml --set workers.celery.serviceAccount.create=true,workers.celery.serviceAccount.name=t
   est-test-test
   ```
   *result:*
   ```yaml
   # Source: airflow/templates/workers/worker-serviceaccount.yaml
   apiVersion: v1
   kind: ServiceAccount
   automountServiceAccountToken: true
   metadata:
     name: "test-test-test"
     labels:
       tier: airflow
       component: worker
       release: test
       chart: "airflow-1.21.0"
       heritage: Helm
   ```
   So the account is created as expected and:
   ```shell
   helm template test chart/ --show-only templates/workers/worker-deployment.yaml --set workers.celery.serviceAccount.create=true,workers.celery.serviceAccount.name=test-test-test | grep serviceAccount
   ```
   ```yaml
   serviceAccountName: "test-test-test"
   ```
   
   ---
   
   As for mechanism implementation clarification:
   
   > All Celery worker templates use the worker.serviceAccountName helper, which reads from workers.serviceAccount (the deprecated path) and never checks workers.celery.serviceAccount. When a user only sets the new path, the helper falls back to generating a default name ({{ .Release.Name }}-worker), ignoring the user's configuration.
   
   In checkes it, but not directly. The key is in the content, which is in context passed to the function `worker.serviceAccountName`. All of the values from `workers.celery.*` override the `workers.*` values with a couple of lines in every worker-related template file. It basically uses the custom merge function to handle that logic. It was done that way to make Workers Sets and deprecations work in #60420. The #64730 introduces the new `worker.kubernetes.serviceAccountName` to fully separate the Kubernetes SA from the Celery SA. Old was retained unchanged to preserve backward compatibility.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] workers.celery.serviceAccount.name is not used by Celery worker templates or RBAC bindings [airflow]

Posted by "benrifkind (via GitHub)" <gi...@apache.org>.
benrifkind commented on issue #67261:
URL: https://github.com/apache/airflow/issues/67261#issuecomment-4503569671

   @Miretpl 
   > could you please add the Helm configuration? It can be hard to find an issue without it, as it is connected to Helm.
   
   For sure. Here's what I have for the workers in values.yaml
   ```yaml
   "workers":
     "celery":
       "keda":
         "enabled": true
         "minReplicaCount": 0
       "persistence":
         "enabled": false
       "podAnnotations":
         "karpenter.sh/do-not-disrupt": "true"
       "replicas": 0
       "resources":
         "limits":
           "cpu": "3"
           "memory": "10Gi"
         "requests":
           "cpu": "2"
           "memory": "8Gi"
       "serviceAccount":
         "name": "worker"
       "terminationGracePeriodSeconds": 1800
   ```
   
   The relevant bit. The service account name is **worker**.
   ```yaml
   "workers":
     "celery":
       "serviceAccount":
         "name": "worker"
   ```
   
   The airflow-pod-launcher-rolebinding sets its worker subject to airflow-worker (the chart's default) instead of worker: 
   ```
   $> kubectl get rolebinding airflow-pod-launcher-rolebinding -n airflow-staging -o jsonpath='{.subjects}'
   [
       {
           "kind": "ServiceAccount",
           "name": "airflow-worker",
           "namespace": "airflow-staging"
       },
       {
           "kind": "ServiceAccount",
           "name": "airflow-triggerer",
           "namespace": "airflow-staging"
       }
   ]
   ```
   
   The service account name on the worker deployment is correct
   ```
   $>. kubectl get deployment airflow-worker  -o jsonpath='{.spec.template.spec.serviceAccountName}'
   ```
   
   This gets me this error when I a k8s task pod runs (via the `KubernetesPodOperator`)
   ```
   [2026-05-20 17:40:25] ERROR - Task failed with exception
   ApiException: (403)
   Reason: Forbidden
   HTTP response headers: HTTPHeaderDict({'Audit-Id': 'b2b92098-e557-49e6-b1e6-765dca6d4723', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '2ebe0f7d-8cbb-4c2d-9851-8f9f32e0007f', 'X-Kubernetes-Pf-Prioritylevel-Uid': '7650eaf2-ef5b-4713-b0df-0abf35cb359f', 'Date': 'Wed, 20 May 2026 23:40:25 GMT', 'Content-Length': '297'})
   HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:airflow-staging:worker\" cannot list resource \"pods\" in API group \"\" in the namespace \"airflow-staging\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
   
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1263 in run
   ```
   
   
   This config solves the issue for me. I think the duplication is unnecessary and I just need the top level serviceAccount key set
   ```yaml
   "workers":
     "celery":
       "serviceAccount":
         "name": "worker"
     "serviceAccount":
       "name": "worker"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org