You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ruffury (via GitHub)" <gi...@apache.org> on 2024/08/12 14:10:03 UTC
[I] Task policy is not setting for retries [airflow]
ruffury opened a new issue, #41405:
URL: https://github.com/apache/airflow/issues/41405
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.9.2
### What happened?
We moved our cluster from KubernetesExecutor to CeleryKubernetesExecutor. We have a lot of dags with `executor_config` parameter. So, we have situation where we have a lot of dags with `executor_config`, which need KubernetesExecutor because it uses custom image. And we have rest of dags without this parameter, so they should execute on the workers.
To achieve this, we create simple `cluster_policy` to task level
```
from airflow.models.baseoperator import BaseOperator
from airflow.policies import hookimpl
@hookimpl
def task_policy(task: BaseOperator):
use_kubernetes_if_config(task=task)
def use_kubernetes_if_config(task: BaseOperator):
if task.executor_config:
task.queue = 'kubernetes'
```
but we have problem - this policy is not working for retries. First run of task is executing in Kubernetes, but retries are executing on the celery and are failing because of lack of libraries (we need custom image there)
### What you think should happen instead?
Retries of task should also execute in Kubernetes pod using kuberneted queue
### How to reproduce
Apply cluster policy from description and dag with task with `executor_config` with custom image
### Operating System
Debian GNU/Linux 12
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.19.0
apache-airflow-providers-celery==3.7.2
apache-airflow-providers-cncf-kubernetes==7.13.0
apache-airflow-providers-common-io==1.3.2
apache-airflow-providers-common-sql==1.14.0
apache-airflow-providers-docker==3.12.0
apache-airflow-providers-elasticsearch==5.4.1
apache-airflow-providers-fab==1.1.1
apache-airflow-providers-ftp==3.9.1
apache-airflow-providers-google==10.19.0
apache-airflow-providers-grpc==3.5.1
apache-airflow-providers-hashicorp==3.7.1
apache-airflow-providers-http==4.11.1
apache-airflow-providers-imap==3.6.1
apache-airflow-providers-microsoft-azure==10.1.1
apache-airflow-providers-mysql==5.6.1
apache-airflow-providers-odbc==4.6.1
apache-airflow-providers-openlineage==1.8.0
apache-airflow-providers-pagerduty==3.7.2
apache-airflow-providers-postgres==5.11.1
apache-airflow-providers-redis==3.7.1
apache-airflow-providers-sendgrid==3.5.1
apache-airflow-providers-sftp==4.10.1
apache-airflow-providers-slack==8.7.1
apache-airflow-providers-smtp==1.7.1
apache-airflow-providers-snowflake==5.5.1
apache-airflow-providers-sqlite==3.8.1
apache-airflow-providers-ssh==3.11.1
apache-airflow-providers-telegram==3.1.1
apache-airflow-providers-trino==5.7.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Executor: CeleryKubernetesExecutor
k8s_version = "1.27"
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task policy is not setting for retries [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #41405:
URL: https://github.com/apache/airflow/issues/41405#issuecomment-5748280088
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task policy is not setting for retries [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on issue #41405:
URL: https://github.com/apache/airflow/issues/41405#issuecomment-2308526488
@RNHTTR going to look at this, bit takes time first time to kubernetes area in airflow, ill try crack it :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task policy is not setting for retries [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on issue #41405:
URL: https://github.com/apache/airflow/issues/41405#issuecomment-3166140093
have you tried this one?
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/cluster-policies.html#how-do-define-a-policy-function
```
def task_instance_mutation_hook(task_instance: TaskInstance):
--
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task policy is not setting for retries [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #41405:
URL: https://github.com/apache/airflow/issues/41405#issuecomment-2797186171
No news. You're more than welcome to take a stab at fixing it though!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task policy is not setting for retries [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #41405: Task policy is not setting for retries
URL: https://github.com/apache/airflow/issues/41405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task policy is not setting for retries [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #41405:
URL: https://github.com/apache/airflow/issues/41405#issuecomment-5352657960
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task policy is not setting for retries [airflow]
Posted by "jdel (via GitHub)" <gi...@apache.org>.
jdel commented on issue #41405:
URL: https://github.com/apache/airflow/issues/41405#issuecomment-2794001962
Hello, any news about this ? It is still happening on 10.4.
If it is the intended behavior, could you please mention it in the documentation: https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/cluster-policies.html#airflow.policies.task_policy
Using it as suggested is really creating un inconsistent behavior:
> You could enforce a specific queue (say the spark queue) for tasks using the SparkOperator to make sure that these tasks get wired to the right workers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task policy is not setting for retries [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #41405:
URL: https://github.com/apache/airflow/issues/41405#issuecomment-2284094288
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task policy is not setting for retries [airflow]
Posted by "deepak4babu (via GitHub)" <gi...@apache.org>.
deepak4babu commented on issue #41405:
URL: https://github.com/apache/airflow/issues/41405#issuecomment-2662845163
Im facing a similar issue in 2.10.3 version.
@RNHTTR @gopidesupavan did you find any fix for this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task policy is not setting for retries [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #41405:
URL: https://github.com/apache/airflow/issues/41405#issuecomment-2322024379
> @RNHTTR going to look at this, bit takes time first time to kubernetes area in airflow, ill try crack it :).
Awesome! I've assigned you. I don't think this actually needs to be running on a Kubernetes cluster to reproduce. You could probably reproduce using a very simple [cluster policy](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/cluster-policies.html#how-do-define-a-policy-function).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org