You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "yquaziii (via GitHub)" <gi...@apache.org> on 2026/07/06 04:55:52 UTC
[PR] Fix CeleryKubernetesExecutor starting Kubernetes watcher client-side [airflow]
yquaziii opened a new pull request, #69430:
URL: https://github.com/apache/airflow/pull/69430
When using CeleryKubernetesExecutor, starting the executor client-side (e.g., during task execution via airflow tasks run) unconditionally initialized the Kubernetes client and spawned pod watchers. This resulted in authorization failures (403 Forbidden) or config errors (ConfigException) on worker pods that lack Kubernetes API access. This change conditionally initializes the Kubernetes client and watchers only if a valid scheduler_job_id is set (indicating a scheduler/backfill job). Closes #49074
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
### Root Cause
When executing airflow tasks run client-side (e.g. on Celery workers), the CLI loads the configured executor. For CeleryKubernetesExecutor, this starts both the celery and kubernetes executors.
However, KubernetesExecutor.start() unconditionally:
Calls get_kube_client() to load config/session.
Instantiates AirflowKubernetesScheduler and immediately spawns KubernetesJobWatcher threads.
This client-side process only queues/runs the task and doesn't need to watch pod status. On workers lacking RBAC permissions (or running out-of-cluster), this leads to immediate crashes.
### Proposed Fix
We can make the initialization of the Kubernetes client and watchers conditional. If self.job_id is not present (indicating a client-side execution path, not a scheduler daemon/backfill job), we can return early from KubernetesExecutor.start() and skip sync() actions.
I have implemented this fix and verified that:
The client-side task run command now succeeds.
All existing KubernetesExecutor and CeleryKubernetesExecutor unit tests pass.
Added a new unit test validating this conditional startup behavior.
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ X ] Yes (please specify the tool below)
- Generated-by: Google Antigravity following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix CeleryKubernetesExecutor starting Kubernetes watcher client-side [airflow]
Posted by "yquaziii (via GitHub)" <gi...@apache.org>.
yquaziii commented on PR #69430:
URL: https://github.com/apache/airflow/pull/69430#issuecomment-4998930093
@hussein-awala, @jedcunningham any update on above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix CeleryKubernetesExecutor starting Kubernetes watcher client-side [airflow]
Posted by "yquaziii (via GitHub)" <gi...@apache.org>.
yquaziii closed pull request #69430: Fix CeleryKubernetesExecutor starting Kubernetes watcher client-side
URL: https://github.com/apache/airflow/pull/69430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix CeleryKubernetesExecutor starting Kubernetes watcher client-side [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69430:
URL: https://github.com/apache/airflow/pull/69430#issuecomment-5486731920
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org