You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kadubhumika (via GitHub)" <gi...@apache.org> on 2026/10/17 10:12:47 UTC

[PR] Feature/on success succeed dagrun [airflow]

kadubhumika opened a new pull request, #74916:
URL: https://github.com/apache/airflow/pull/74916

   <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   Add `on_success_succeed_dagrun` to teardown tasks, so that a successful teardown
   no longer hides a failed work task.
   
   closes: #74727
   
   **Problem**
   
   With `@teardown(on_failure_fail_dagrun=True)` the teardown is the only task that
   decides the Dag run state. If a work task fails but the teardown succeeds, the
   Dag run is marked SUCCESS and the failure is hidden.
   
   **Fix**
   
   New teardown option `on_success_succeed_dagrun` (default `True`, so existing Dags
   behave exactly as before). With `on_failure_fail_dagrun=True` and
   `on_success_succeed_dagrun=False`:
   
   | Work    | Teardown | Dag run |
   |---------|----------|---------|
   | failed  | success  | FAILED  |
   | success | failed   | FAILED  |
   | success | success  | SUCCESS |
   
   **What changed**
   
   - `airflow-core/.../models/dagrun.py`: `_tis_for_dagrun_state` also evaluates the
     tasks before a teardown that sets `on_success_succeed_dagrun=False`.
   - `task-sdk` (`@teardown`, `as_teardown()`, `_internal/abstractoperator.py`,
     `_internal/mixins.py`, `xcom_arg.py`, `bases/decorator.py`, `bases/operator.py`,
     `mappedoperator.py`): new argument, property and validation, with support for
     mapped tasks and `.override()`.
   - `airflow-core/.../serialization/` (`schema.json`, `serialized_objects.py`,
     `definitions/baseoperator.py`, `definitions/mappedoperator.py`,
     `dag_version_diff.py`): the new field is serialized and deserialized.
   - Tests: `test_dagrun.py` covers all six flag and state combinations;
     `test_dag_serialization.py` registers the new serialized field.
   - Newsfragment added.
   
   **Testing**
   
   `test_dagrun.py`, `test_dag_serialization.py` and the task-sdk
   `test_setup_t


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feature/on success succeed dagrun [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on code in PR #74916:
URL: https://github.com/apache/airflow/pull/74916#discussion_r4302541199


##########
providers/google/src/airflow/providers/google/cloud/hooks/dataproc.py:
##########
@@ -26,7 +26,7 @@
 from collections.abc import MutableSequence, Sequence

Review Comment:
   Hi @shahar1 , thank you for reviewing my PR!
   
   I've addressed your feedback:
   - Removed the unrelated Dataproc commits (my branch was created from the wrong
     base). The PR now contains only the teardown change.
   - Added the how-to docs in setup-and-teardown.rst.
   - Added task-sdk tests for the new argument.
   - Fixed the CI failures: renamed the newsfragment to the PR number, synced the
     Go/Java/TypeScript schema copies, and excluded the teardown-only flag from the
     Go/Java/TS SDK generators, the same way on_failure_fail_dagrun is excluded.
   - Fixed the title and added the AI disclosure to the PR description.
   Could you please review again when you have time? Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feature/on success succeed dagrun [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on PR #74916:
URL: https://github.com/apache/airflow/pull/74916#issuecomment-6233559698

   Thank You ✨
   
   On Sat, Oct 17, 2026, 5:20 PM Shahar Epstein ***@***.***>
   wrote:
   
   > ***@***.**** commented on this pull request.
   > ------------------------------
   >
   > In providers/google/src/airflow/providers/google/cloud/hooks/dataproc.py
   > <https://github.com/apache/airflow/pull/74916#discussion_r4302556526>:
   >
   > > @@ -26,7 +26,7 @@
   >  from collections.abc import MutableSequence, Sequence
   >
   > Great! I'll take a look after CI is done
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/airflow/pull/74916?email_source=notifications&email_token=BMB777KZMHUVRDMWGBZDBN35UNMJPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKNJVG43DGNBWHA22M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#discussion_r4302556526>,
   > or unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/BMB777LAFAFZHHVCKUWJSAL5UNMJPAVCNFSNUABEKJSXA33TNF2G64TZHMZTGOBYGQ4DSMJ3JFZXG5LFHM2TQOJXGMYDCMBZG6QXMAQ>
   > .
   > Triage notifications, keep track of coding agent tasks and review pull
   > requests on the go with GitHub Mobile for iOS
   > <https://github.com/notifications/mobile/ios/BMB777N6LUROVEAXDMDB7EL5UNMJPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKNJVG43DGNBWHA22M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG>
   > and Android
   > <https://github.com/notifications/mobile/android/BMB777N7Z2RDCFPA42KMZCL5UNMJPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKNJVG43DGNBWHA22M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>.
   > Download it today!
   > You are receiving this because you authored the thread.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feature/on success succeed dagrun [airflow]

Posted by "yuseok89 (via GitHub)" <gi...@apache.org>.
yuseok89 commented on code in PR #74916:
URL: https://github.com/apache/airflow/pull/74916#discussion_r4305977141


##########
airflow-core/tests/unit/models/test_dagrun.py:
##########


Review Comment:
   For a teardown declared with `on_failure_fail_dagrun=True`, has applying `.as_teardown(on_success_succeed_dagrun=False)` after `.expand()` also been checked?
   
   If this path isn't covered yet, would it make sense to add a test checking that
   
   - the configured value is preserved after the Dag is serialized and deserialized
   - the restored value is reflected in the Dag run state evaluation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feature/on success succeed dagrun [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #74916:
URL: https://github.com/apache/airflow/pull/74916#discussion_r4302454167


##########
providers/google/src/airflow/providers/google/cloud/hooks/dataproc.py:
##########
@@ -26,7 +26,7 @@
 from collections.abc import MutableSequence, Sequence

Review Comment:
   It seems like you might have accidentally mixed up commits from another branch - please undo unrelated changes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feature/on success succeed dagrun [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #74916:
URL: https://github.com/apache/airflow/pull/74916#discussion_r4302556526


##########
providers/google/src/airflow/providers/google/cloud/hooks/dataproc.py:
##########
@@ -26,7 +26,7 @@
 from collections.abc import MutableSequence, Sequence

Review Comment:
   Great! I'll take a look after CI is done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `on_success_succeed_dagrun` option to teardown tasks [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on PR #74916:
URL: https://github.com/apache/airflow/pull/74916#issuecomment-6397126204

   CI passed has been waiting for a while, could a maintainer please re-trigger the workflows?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `on_success_succeed_dagrun` option to teardown tasks [airflow]

Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #74916:
URL: https://github.com/apache/airflow/pull/74916#discussion_r4344247728


##########
airflow-core/src/airflow/serialization/serialized_objects.py:
##########
@@ -1222,6 +1222,8 @@ def populate_operator(
                 v = cls.deserialize(v)
             elif k == "_on_failure_fail_dagrun":
                 k = "on_failure_fail_dagrun"
+            elif k == "_on_success_succeed_dagrun":
+                k = "on_success_succeed_dagrun"

Review Comment:
   nit: the `elif k == "_on_success_succeed_dagrun"` branch never runs. `_on_success_succeed_dagrun` is not in `op.get_serialized_fields()`, so the earlier `k not in op.get_serialized_fields()` branch handles the key first.
   
   Whether this `elif` branch is needed depends on how the fix for `.expand().as_teardown(...)` in https://github.com/apache/airflow/pull/74916#discussion_r4305977141 is written. If that fix stores the `on_success_succeed_dagrun` flag in `partial_kwargs`, this branch stays unused. If that fix instead writes `_on_success_succeed_dagrun` as a top-level key of mapped tasks, this branch is needed. In that case, `_on_success_succeed_dagrun` must also be added to `SerializedMappedOperator.get_serialized_fields()`. Otherwise the earlier `k not in op.get_serialized_fields()` branch still takes the `_on_success_succeed_dagrun` key first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feature/on success succeed dagrun [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on code in PR #74916:
URL: https://github.com/apache/airflow/pull/74916#discussion_r4302486852


##########
providers/google/src/airflow/providers/google/cloud/hooks/dataproc.py:
##########
@@ -26,7 +26,7 @@
 from collections.abc import MutableSequence, Sequence

Review Comment:
   yes sure sorry for this i will soon change it.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Feature/on success succeed dagrun [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on code in PR #74916:
URL: https://github.com/apache/airflow/pull/74916#discussion_r4302541199


##########
providers/google/src/airflow/providers/google/cloud/hooks/dataproc.py:
##########
@@ -26,7 +26,7 @@
 from collections.abc import MutableSequence, Sequence

Review Comment:
   Hi @shahar1 , thank you for reviewing my PR!
   
   I've addressed your feedback:
   - Removed the unrelated Dataproc commits (my branch was created from the wrong
     base). The PR now contains only the teardown change.
   - Added the how-to docs in setup andte ardown.rst.
   - Added task-sdk tests for the new argument.
   - Fixed the CI failures: renamed the newsfragment to the PR number, synced the
     Go/Java/TypeScript schema copies, and excluded the teardown-only flag from the
     Go/Java/TS SDK generators, the same way on_failure_fail_dagrun is excluded.
   - Fixed the title and added the AI disclosure to the PR description.
   Could you please review again when you have time? Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `on_success_succeed_dagrun` option to teardown tasks [airflow]

Posted by "kadubhumika (via GitHub)" <gi...@apache.org>.
kadubhumika commented on code in PR #74916:
URL: https://github.com/apache/airflow/pull/74916#discussion_r4352401974


##########
airflow-core/src/airflow/serialization/serialized_objects.py:
##########
@@ -1222,6 +1222,8 @@ def populate_operator(
                 v = cls.deserialize(v)
             elif k == "_on_failure_fail_dagrun":
                 k = "on_failure_fail_dagrun"
+            elif k == "_on_success_succeed_dagrun":
+                k = "on_success_succeed_dagrun"

Review Comment:
   @FrankYang0529 Thanks for pointing this out. The mapped operator stores `on_success_succeed_dagrun` in `partial_kwargs`, so I removed the redundant `_on_success_succeed_dagrun` rename branch and left the existing failure-flag handling unchanged. I also extended `test_teardown_mapped_serialization` to verify that `on_success_succeed_dagrun=False` is preserved through serialization and deserialization. The targeted test passes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add `on_success_succeed_dagrun` option to teardown tasks [airflow]

Posted by "shahar1 (via GitHub) via commits" <co...@airflow.apache.org>.
shahar1 commented on code in PR #74916:
URL: https://github.com/apache/airflow/pull/74916#discussion_r4500477194


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1201,17 +1201,34 @@ def _tis_for_dagrun_state(self, *, dag, tis):
         users may enable such consideration with on_failure_fail_dagrun.
         """
 
-        def is_effective_leaf(task):
-            for down_task_id in task.downstream_task_ids:
-                down_task = dag.get_task(down_task_id)
-                if not down_task.is_teardown or down_task.on_failure_fail_dagrun:
-                    # we found a down task that is not ignorable; not a leaf
-                    return False
-            # we found no ignorable downstreams
-            # evaluate whether task is itself ignorable
+        def counts_for_state(task):
             return not task.is_teardown or task.on_failure_fail_dagrun
 
-        leaf_task_ids = {x.task_id for x in dag.tasks if is_effective_leaf(x)}
+        def effective_leaf_ids(is_counted):
+            def is_effective_leaf(task):
+                for down_task_id in task.downstream_task_ids:
+                    if is_counted(dag.get_task(down_task_id)):
+                        # we found a down task that is not ignorable; not a leaf
+                        return False
+                # we found no ignorable downstreams
+                # evaluate whether task is itself ignorable
+                return is_counted(task)
+
+            return {x.task_id for x in dag.tasks if is_effective_leaf(x)}
+
+        leaf_task_ids = effective_leaf_ids(counts_for_state)
+
+        # A teardown with on_failure_fail_dagrun=True and on_success_succeed_dagrun=False must not
+        # hide failures of the tasks before it, so those tasks count too (as if it were ignorable).
+        strict_teardowns = {

Review Comment:
   **minor**
   
   This does not cover a work task with two teardowns that have different settings:
   
   ```python
   w >> strict_td()   # on_failure_fail_dagrun=True, on_success_succeed_dagrun=False
   w >> lenient_td()  # on_failure_fail_dagrun=True
   ```
   
   With `work` failed and both teardowns successful, the run is `success`, because `lenient_td` still counts in the second pass and `work` is not a leaf. The comment above says a strict teardown "must not hide failures of the tasks before it". Which one should win here? If the strict teardown should, this needs a fix and a test. If not, the docs should state the limit.
   



##########
task-sdk/src/airflow/sdk/definitions/mappedoperator.py:
##########
@@ -375,6 +375,7 @@ def get_serialized_fields(cls):
             "_is_setup",
             "_is_teardown",
             "_on_failure_fail_dagrun",
+            "_on_success_succeed_dagrun",

Review Comment:
   **major**
   
   `.as_teardown()` after `.expand()` loses this flag:
   
   ```python
   @teardown(on_failure_fail_dagrun=True)
   def cleanup(val): ...
   
   t = cleanup.expand(val=[1, 2]).as_teardown(on_success_succeed_dagrun=False)
   work() >> t
   ```
   
   With `work` failed and `cleanup` successful at 945c69881d, the deserialized task has `on_success_succeed_dagrun=True` and the Dag run is `success`.
   
   `MappedOperator` inherits the `AbstractOperator` property, which writes `_on_success_succeed_dagrun`, but mapped serialization only reads `partial_kwargs`. So the value reaches `partial_kwargs` only when it is passed to the decorator or `.override()`, not through `.as_teardown()`.
   
   Adding a property next to `is_teardown` fixed this case when I tried it (the run became `failed`):
   
   ```python
   @property
   def on_success_succeed_dagrun(self) -> bool:
       return self.partial_kwargs.get("on_success_succeed_dagrun", True)
   
   @on_success_succeed_dagrun.setter
   def on_success_succeed_dagrun(self, value: bool) -> None:
       if value is False and self.is_teardown is not True:
           raise ValueError(
               f"Cannot set task on_success_succeed_dagrun for "
               f"'{self.task_id}' because it is not a teardown task."
           )
       self.partial_kwargs["on_success_succeed_dagrun"] = value
   ```
   
   Classic mapped teardowns need one more decision. `PythonOperator.partial(...).expand(...).as_teardown(on_failure_fail_dagrun=True, on_success_succeed_dagrun=False)` also loses `on_failure_fail_dagrun`, through the same gap, in code this PR does not change. Even with the property above, `work` successful and the teardown failed gives a `success` run. Fixing that changes existing behaviour (the docstring of `test_one_to_many_as_teardown_on_failure_fail_dagrun` expects a successful run), so it may belong in its own PR. Until then, the docs should say this form is not supported.
   
   Please add a test for `.expand().as_teardown(...)` that covers the serialization round trip and the Dag run state.
   



##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -4750,6 +4750,47 @@ def mytask():
     assert dr.state == dag_run_state
 
 
+@pytest.mark.parametrize(
+    ("on_success_succeed_dagrun", "work_state", "teardown_state", "expected"),
+    [
+        (True, State.FAILED, State.SUCCESS, DagRunState.SUCCESS),
+        (True, State.SUCCESS, State.FAILED, DagRunState.FAILED),
+        (True, State.SUCCESS, State.SUCCESS, DagRunState.SUCCESS),
+        (False, State.FAILED, State.SUCCESS, DagRunState.FAILED),
+        (False, State.SUCCESS, State.FAILED, DagRunState.FAILED),
+        (False, State.SUCCESS, State.SUCCESS, DagRunState.SUCCESS),
+    ],
+)
+def test_teardown_on_success_succeed_dagrun(

Review Comment:
   **minor**
   
   From `.github/instructions/code-review.instructions.md`:
   
   > **Flag any changed or added behaviour without a corresponding test, and flag tests that a reviewer cannot fail by reverting the PR's change.**
   
   These changed paths have no test: Dag run state for a mapped teardown (`SerializedMappedOperator.on_success_succeed_dagrun`), `.override(on_success_succeed_dagrun=False)` and `MappedOperator.unmap()`. The `.override()` and `unmap()` paths gave the right result when I ran them, so only the tests are missing.
   



##########
airflow-core/newsfragments/74916.feature.rst:
##########
@@ -0,0 +1 @@
+Add ``on_success_succeed_dagrun`` to teardown tasks. With ``on_failure_fail_dagrun=True`` and ``on_success_succeed_dagrun=False``, a successful teardown no longer hides a failed task.

Review Comment:
   **minor**
   
   From `AGENTS.md`:
   
   > In `airflow-core/` use only `significant` newsfragments, and only for a change users must read before upgrading: a behavior change, a removal or a new default, or migration steps.
   
   > Never add one for an ordinary bug fix, feature, improvement or doc change
   
   This is an opt-in feature with an unchanged default, so please delete this file.
   



##########
airflow-core/docs/howto/setup-and-teardown.rst:
##########
@@ -146,6 +146,21 @@ Continuing with the example above, if you want the run's success to depend on ``
 
   create_cluster >> run_query >> delete_cluster.as_teardown(setups=create_cluster, on_failure_fail_dagrun=True)
 
+With ``on_failure_fail_dagrun=True`` the teardown alone decides the Dag run state: if ``run_query`` fails but
+``delete_cluster`` succeeds, the run is marked successful. To also keep failures of the tasks before the
+teardown, set ``on_success_succeed_dagrun=False``. The run then fails when the teardown fails **or** when any

Review Comment:
   **minor**
   
   "any task before it failed" promises more than the code does. With `first` failed, `second` (`trigger_rule="all_done"`) successful and `cleanup` successful, the run is `success`. Only the last tasks before the teardown count, as in a Dag without one, and that behaviour looks reasonable to me. Please narrow the wording, for example "when one of the last tasks before it fails". The property docstring in `abstractoperator.py:196` ("failed tasks upstream of the teardown") has the same problem.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org