You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "lezaf (via GitHub)" <gi...@apache.org> on 2025/12/08 22:12:39 UTC
[D] Orchestration of multi-step workflow failures handlers [airflow]
GitHub user lezaf created a discussion: Orchestration of multi-step workflow failures handlers
Hey everyone, 👋
I’m working on a dynamic Airflow DAG where each item goes through a sequence of mapped tasks inside a task group (e.g., step_1 -> step_2 -> step_3). Each step may fail for specific items, and when that happens I want to route the original item to a corresponding failure handler to e.g. perform some compensation steps. Below is an example of a desired DAG outline:
<img width="1077" height="389" alt="image" src="https://github.com/user-attachments/assets/b4936218-eb24-4fb4-908f-3c604ba82156" />
And a part of the underlying code for simplicity:
```py
@task_group
def processing_group(item):
@task
def step_1(item):
if should_fail(item):
raise ValueError("step 1 failed")
return item
@task(trigger_rule="all_failed")
def step_1_fail_handler(step1_output):
# <--- I want this task to receive the original input
pass
# rest of steps implentation
step1_res = step_1(item)
step1_failed = step_1_fail_handler(step1_res)
# rest of steps wiring
return {
"step1_failed": step1_failed,
"step2_failed": step2_failed,
"step3_failed": step3_failed,
"step3_success": step3_res,
}
```
## Problem
When the mapped task step_1 fails, Airflow does not return any output, so the failure handler cannot access the original (or even better enriched as per requirements) item via the step result.
I have concluded to a workaround using _current context_ and _xcom_push_, something like:
```py
def processing_group(item):
@task
def step_1(item):
if should_fail(item):
# ============ Context manipulation (start) ============
context = get_current_context()
context['ti'].xcom_push(
key='return_value',
value={
**item,
"step_1_failed_flag": True,
"error": "An error message",
}
)
# ============ Context manipulation (end) ============
raise ValueError("step 1 failed")
return item
step1_res = step_1(item)
# rest
return {
"step1_res": step1_res
"step2_res": step2_res
"step3_res": step3_res,
}
```
This way, I can aggregate on failure flags and achieve my goal. But it easily gets complicated and hard to maintain.
## My question:
Is there an Airflow native way to handle this situation, or this idiomatic pattern is the solution here?
Thanks in advance for any guidance! 🙏
GitHub link: https://github.com/apache/airflow/discussions/59221
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Orchestration of multi-step workflow failures handlers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Orchestration of multi-step workflow failures handlers
No - there is no other way currently. But it would likely be a good idea to introduce a special exception that could take result as parameter. Feel free to create a PR or open an issue for it.
GitHub link: https://github.com/apache/airflow/discussions/59221#discussioncomment-15202820
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Orchestration of multi-step workflow failures handlers [airflow]
Posted by "lezaf (via GitHub)" <gi...@apache.org>.
GitHub user lezaf added a comment to the discussion: Orchestration of multi-step workflow failures handlers
Ok, that’s interesting. I’ll open an issue for this, having a way to attach a result or payload to a task failure would be very useful for cases like mine, where each mapped item needs follow-up handling and final aggregation.
Thanks a lot!
GitHub link: https://github.com/apache/airflow/discussions/59221#discussioncomment-15202871
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org