You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jscheffl (via GitHub)" <gi...@apache.org> on 2025/01/09 22:27:35 UTC
[PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
jscheffl opened a new pull request, #45530:
URL: https://github.com/apache/airflow/pull/45530
Related to Slack topic: https://apache-airflow.slack.com/archives/CCR6P6JRL/p1736440079894049
We noticed that if a DAG run is set to failed, all tasks are either set to failed or skipped. But if Teardown Tasks are used in a DAG, they are not executed. This could lead to infrastructure or external dependencies not properly cleaned-up.
This PR changes the behavior and does NOT fail/skip teardown tasks if a DAG is set to failed.
A side effect as consequence might be that the DAG is after the call NOT failed, else if it would set it to failed, then any teardown task (even if not skipped/failed) will not scheduled anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #45530:
URL: https://github.com/apache/airflow/pull/45530#issuecomment-2585344164
> Nice catch! One edge case that I thought about though* - what if the corresponding setup task hasn't finished running yet? (if such exists ofc) For example:  If you set the DAG run to `failed` before the c
luster was created, the `delete_cluster` should be skipped.
>
> * Maybe the current architecture already handles, but it's worth checking before merging this PR.
Oh, yeah. Tested this explicitly. And actually the Scheduler had this feature before (added this to docs as well, was also new to me): If the setup task is skipped (which is the state when you mark as failed and it was not running) then the teardown is also skipped,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl merged PR #45530:
URL: https://github.com/apache/airflow/pull/45530
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "rakesh-shah (via GitHub)" <gi...@apache.org>.
rakesh-shah commented on PR #45530:
URL: https://github.com/apache/airflow/pull/45530#issuecomment-3060495743
So @jscheffl in which version of Airflow this issue got fixed as we are using 2.7.2 and we can see an issue where sometime when a task fails I am guessing it because of Task Kill might be due to Pod crash does not call failure callback could you please help me on this if this issue was also fixed or should I raise one.
thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #45530:
URL: https://github.com/apache/airflow/pull/45530#issuecomment-2585344311
Added some docs as I thought about that slightly the behavior changes - to ensure it is properly documented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45530:
URL: https://github.com/apache/airflow/pull/45530#issuecomment-2585356465
### Backport successfully created: v2-10-test
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v2-10-test</td>
<td><a href="https://github.com/apache/airflow/pull/45581"><img src="https://img.shields.io/badge/PR-45581-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #45530:
URL: https://github.com/apache/airflow/pull/45530#issuecomment-2590403975
> just notice this could potentially be a breaking change, but doesn't look like something we need a migration rule 🤔 would like to confirm whether we might want to have such a rule?
I'd say it's a bugfix. Not all "change behaviour" is "breaking change". Every bugfix is a "behavioural change" in essence, the important thing is what is the feature intention and whether the change is "fixing" things or "changing intention on how things should work". SemVer is all about **intentions** and not about whether behaviour changes or not - see https://semver.org/
In this case it seems reasonable to assume that this behaviour was intention of the "teardown" behaviour - so technically it's a bugfix (and it's been already backported to 2.10 it seems.
It might require a newsfragment, but IMHO - that's more than enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #45530:
URL: https://github.com/apache/airflow/pull/45530#issuecomment-3060641122
> So @jscheffl we are using 2.7.2 and we can see an issue where sometime when a task fails I am guessing it because of Task Kill might be due to Pod crash does not call failure callback could you please help me on this if this issue was also fixed or should I raise one. thanks
This was fixed in 2.10.5 - See milestone flag on the right side. Also Release Notes help in this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "rakesh-shah (via GitHub)" <gi...@apache.org>.
rakesh-shah commented on PR #45530:
URL: https://github.com/apache/airflow/pull/45530#issuecomment-3061921243
> > So @jscheffl we are using 2.7.2 and we can see an issue where sometime when a task fails I am guessing it because of Task Kill might be due to Pod crash does not call failure callback could you please help me on this if this issue was also fixed or should I raise one. thanks
>
> This was fixed in 2.10.5 - See milestone flag on the right side. Also Release Notes help in this
Yeah thanks for that @jscheffl I saw the release notes and it says it will work for teardown when the state is changed manually to failed
But @jscheffl I was thinking if it also covers when the issue appears due to Kill event during a pod crash which generates a SIGTERM ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "rakesh-shah (via GitHub)" <gi...@apache.org>.
rakesh-shah commented on PR #45530:
URL: https://github.com/apache/airflow/pull/45530#issuecomment-3067825863
> > But @jscheffl I was thinking if it also covers when the issue appears due to Kill event during a pod crash which generates a SIGTERM ?
>
> This sounds like a different use case. But you can test it and if you think it is a bug then please open a new issue for this as it seems to be unrelated. But please test with 2.11.0 or most recent 3.0
Sure @jscheffl sounds good will test it and revert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #45530:
URL: https://github.com/apache/airflow/pull/45530#issuecomment-2588463039
just notice this could potentially be a breaking change, but doesn't look like something we need a migration rule 🤔 would like to confirm whether we might want to have such a rule?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #45530:
URL: https://github.com/apache/airflow/pull/45530#issuecomment-2591027043
> just notice this could potentially be a breaking change, but doesn't look like something we need a migration rule 🤔 would like to confirm whether we might want to have such a rule?
For sure it does not need migration rules. No code change needed.
Yeah and thought also a bit about if it is breaking, asked in #random / Slack for feedback and nobody except @dstandish was responding. Then after passing... I think it is really in the intend for why we have Setup/Teardown. Behaviour change is "just" because of the side effect that the DAG is not immediately failed if you set it to failed as Teardown need to be scheduled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure teardown tasks are executed when DAG run is set to failed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #45530:
URL: https://github.com/apache/airflow/pull/45530#issuecomment-3063763533
> But @jscheffl I was thinking if it also covers when the issue appears due to Kill event during a pod crash which generates a SIGTERM ?
This sounds like a different use case. But you can test it and if you think it is a bug then please open a new issue for this as it seems to be unrelated. But please test with 2.11.0 or most recent 3.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org