You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "will-byrne-cardano (via GitHub)" <gi...@apache.org> on 2023/08/10 15:55:26 UTC
[GitHub] [airflow] will-byrne-cardano commented on issue #26760: DAG on_failure_callback uses wrong context
will-byrne-cardano commented on issue #26760:
URL: https://github.com/apache/airflow/issues/26760#issuecomment-1673495862
> Hi All! Hi @josh-fell, If the same callback is applied to all tasks, I think, it would be triggered by each task failure instead of being executed only once at the DAG failure. Please share your thoughts.
>
> > If you want to gain the context from the failed task, you can set [`on_failure_callback` at the task level](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/baseoperator/index.html?#airflow.models.baseoperator.BaseOperator) or, I presume you'd like the same callback to apply to all tasks, within [`default_args`](https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html?highlight=default_args#default-arguments).
>
> It would be nice to be capable of getting the earliest failed task instance from the context or even all the task instances that failed.
>
> Please advise.
Hi @cklimkowski
One possible workaround you could use that I have employed in some of my dags to find "the earliest failed task instance from the context or even all the task instances that failed" is to use the following pattern when designing callback functions:
```python
def failure_callback(context):
dag_run = context.get("dag_run")
tis = dag_run.get_task_instances()
failed_instances = [ti for ti in tis if ti.state == "failed"]
failed_instance_ids [ti.task_id for ti in failed_instances]
... # add further logic e.g. take first element for first failed task
```
This should always get at least the first failed task instance. However, if a failed task triggers a dag failure and there are still other tasks running, and those tasks end up failing, it won't pick these up as their state will be `running` when the callback fires. A potential workaround (that I've not tested) if you need all failed is to implement a retry with a backoff on your callback that waits until no tasks have the state `running`. This could lead to complications with undead tasks though, so be sure to set some limit on number of retries if you are to do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org