You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "siddjellali (via GitHub)" <gi...@apache.org> on 2025/09/23 17:14:41 UTC
[I] KubernetesExecutor Worker Configuration in Airflow 3.0 [airflow]
siddjellali opened a new issue, #56008:
URL: https://github.com/apache/airflow/issues/56008
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
With Airflow 3.0 and the new Task SDK architecture, the airflow worker command no longer exists, but there's insufficient documentation on how to properly configure worker images for KubernetesExecutor environments using the CNCF Kubernetes provider.
## Environment
- Airflow Version: 3.0.6
- Provider Version: apache-airflow-providers-cncf-kubernetes>=8.1.0
- Python Version: 3.12
- Executor: KubernetesExecutor
- Deployment: Kubernetes
- Operating System: Linux (Docker containers)
## Background
We are migrating from Airflow 2.x to 3.0 and need clarification on the proper way to configure worker containers when using KubernetesExecutor from the CNCF Kubernetes provider, especially given the new Task SDK and API server architecture changes in Airflow 3.0.
## Breaking Change: airflow worker Command Removed in Airflow 3.0
The airflow worker command that existed in Airflow 2.x has been completely removed in Airflow 3.0 due to the new Task SDK architecture (AIP-72). In the previous architecture, workers were long-running processes that connected directly to the metadata database. With Airflow 3.0, the architecture has shifted to a client-server model where workers communicate through a dedicated API server instead of direct database access. For KubernetesExecutor specifically, this means worker pods no longer run persistent worker processes - instead, the KubernetesExecutor directly injects task execution commands (like airflow tasks test) into ephemeral pods that execute individual tasks and terminate. This architectural change improves security by removing direct database access from workers and enables better isolation, but requires updating worker image configurations to remove any references to the obsolete airflow worker command.
“Is apache-airflow-providers-cncf-kubernetes compatible? Do you have an example you could share?”
### Apache Airflow version
3.0.6
### Operating System
Linux
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
_No response_
### What you think should happen instead
_No response_
### How to reproduce
N/A
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesExecutor Worker Configuration in Airflow 3.0 [airflow]
Posted by "siddjellali (via GitHub)" <gi...@apache.org>.
siddjellali commented on issue #56008:
URL: https://github.com/apache/airflow/issues/56008#issuecomment-3325395601
I’ve finalized a solution: I created an entrypoint for my Dockerfile and rebuilt the worker.
What do you think?
```
#!/bin/bash
set -e
echo "🚀 Starting Airflow 3.0 component: ${AIRFLOW_COMPONENT_TYPE}"
wait_for_db() {
echo "⏳ Waiting for database connection..."
timeout 30 bash -c 'until airflow db check; do sleep 1; done'
echo "✅ Database is ready"
}
case "${AIRFLOW_COMPONENT_TYPE}" in
"api-server")
echo "🌐 Starting Airflow 3.0 API Server..."
wait_for_db
airflow db migrate
exec airflow api-server
;;
"scheduler")
echo "📅 Starting Airflow 3.0 Scheduler..."
wait_for_db
exec airflow scheduler
;;
"worker")
echo "🔧 Worker mode - Airflow 3.0"
echo "ℹ️ In Airflow 3.0, workers are managed by KubernetesExecutor"
echo "ℹ️ Task command will be injected automatically"
# Configuration spécifique worker
export AIRFLOW__CORE__EXECUTOR=LocalExecutor
export AIRFLOW__CORE__LOAD_EXAMPLES=False
export PYTHONUNBUFFERED=1
# Le KubernetesExecutor injecte la commande de tâche
# Exemple : airflow tasks test dag_id task_id execution_date
exec "$@"
;;
esac
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesExecutor Worker Configuration in Airflow 3.0 [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #56008: KubernetesExecutor Worker Configuration in Airflow 3.0
URL: https://github.com/apache/airflow/issues/56008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesExecutor Worker Configuration in Airflow 3.0 [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #56008:
URL: https://github.com/apache/airflow/issues/56008#issuecomment-3324899742
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesExecutor Worker Configuration in Airflow 3.0 [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #56008:
URL: https://github.com/apache/airflow/issues/56008#issuecomment-5809424900
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesExecutor Worker Configuration in Airflow 3.0 [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #56008:
URL: https://github.com/apache/airflow/issues/56008#issuecomment-6394676688
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org