You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "namelessjon (via GitHub)" <gi...@apache.org> on 2024/11/26 14:51:29 UTC
[D] Only retry KubernetesPodOperator if the pod scheduling fails [airflow]
GitHub user namelessjon created a discussion: Only retry KubernetesPodOperator if the pod scheduling fails
Hi
I have a bunch of jobs using the KubernetesPodOperator (via EKS, but that doesn't seem relevant to this discussion). Pod scheduling in our cluster is a little unreliable, so I set retries for the operator via airflow to avoid transient failures. So far, so good.
However, independently of this, the jobs running in the pods sometimes fail (generally due to bad inputs somewhere upstream, but regardless of the reason, these failures are very unlikely to resolve via trying again). Due to the retries policy, they will still retry, which delays notifications of errors and results in pointless effort in running the task.
How can I configure things so that if the pod fails to schedule it will be retried, but if the task runs to completion, a non-zero exit code will be treated as if an `AirflowFailException` was raised?
Suggestions gratefully accepted!
GitHub link: https://github.com/apache/airflow/discussions/44390
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Only retry KubernetesPodOperator if the pod scheduling fails [airflow]
Posted by "wolvery (via GitHub)" <gi...@apache.org>.
GitHub user wolvery added a comment to the discussion: Only retry KubernetesPodOperator if the pod scheduling fails
Hey! I'm interested in this as well. Any finding on this topic? I think it should have a PodFailurePolicy callback function
GitHub link: https://github.com/apache/airflow/discussions/44390#discussioncomment-13315103
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Only retry KubernetesPodOperator if the pod scheduling fails [airflow]
Posted by "wolvery (via GitHub)" <gi...@apache.org>.
GitHub user wolvery added a comment to the discussion: Only retry KubernetesPodOperator if the pod scheduling fails
Hey, we can use the skip_on_exit_code:
https://github.com/apache/airflow/pull/33702
GitHub link: https://github.com/apache/airflow/discussions/44390#discussioncomment-13322028
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Only retry KubernetesPodOperator if the pod scheduling fails [airflow]
Posted by "namelessjon (via GitHub)" <gi...@apache.org>.
GitHub user namelessjon edited a discussion: Only retry KubernetesPodOperator if the pod scheduling fails
Hi
I have a bunch of jobs using the KubernetesPodOperator (via EKS, but that doesn't seem relevant to this discussion). Pod scheduling in our cluster is a little unreliable, so I set retries for the operator via airflow to avoid transient failures. So far, so good.
However, independently of this, the jobs running in the pods sometimes fail (generally due to bad inputs somewhere upstream, but regardless of the reason, these failures are very unlikely to resolve via trying again). Due to the retries policy, they will still retry, which delays notifications of errors and results in pointless effort in running the task.
How can I configure things so that if the pod fails to schedule it will be retried, but if the task runs to completion, a non-zero exit code will be treated as if an `AirflowFailException` was raised?
Suggestions gratefully accepted!
EDIT: from some investigation, it seems the handling of the exit code all happens in the [https://github.com/apache/airflow/blob/main/providers/src/airflow/providers/cncf/kubernetes/operators/pod.py#L851](cleanup) method, with no real option to interfere. But is that correct?
GitHub link: https://github.com/apache/airflow/discussions/44390
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Only retry KubernetesPodOperator if the pod scheduling fails [airflow]
Posted by "namelessjon (via GitHub)" <gi...@apache.org>.
GitHub user namelessjon edited a discussion: Only retry KubernetesPodOperator if the pod scheduling fails
Hi
I have a bunch of jobs using the KubernetesPodOperator (via EKS, but that doesn't seem relevant to this discussion). Pod scheduling in our cluster is a little unreliable, so I set retries for the operator via airflow to avoid transient failures. So far, so good.
However, independently of this, the jobs running in the pods sometimes fail (generally due to bad inputs somewhere upstream, but regardless of the reason, these failures are very unlikely to resolve via trying again). Due to the retries policy, they will still retry, which delays notifications of errors and results in pointless effort in running the task.
How can I configure things so that if the pod fails to schedule it will be retried, but if the task runs to completion, a non-zero exit code will be treated as if an `AirflowFailException` was raised?
Suggestions gratefully accepted!
EDIT: from some investigation, it seems the handling of the exit code all happens in the [cleanup](https://github.com/apache/airflow/blob/main/providers/src/airflow/providers/cncf/kubernetes/operators/pod.py#L851) method, with no real option to interfere. But is that correct?
GitHub link: https://github.com/apache/airflow/discussions/44390
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org