You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "yuseok89 (via GitHub)" <gi...@apache.org> on 2026/06/06 14:56:52 UTC
[PR] Add TaskGroup-level retries to retry a whole group as a unit (WIP) [airflow]
yuseok89 opened a new pull request, #68139:
URL: https://github.com/apache/airflow/pull/68139
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
WIP
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [x] Yes (please specify the tool below)
- Opus 4.8
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TaskGroup-level retries to retry a whole group as a unit [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #68139:
URL: https://github.com/apache/airflow/pull/68139#issuecomment-5839300847
Hello @yuseok89 - thank you for your contributions to Apache Airflow!
The Airflow community has introduced a limit of **5 open pull requests at a time** for contributors without write access to the repository. You currently have 11 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
* #65323 - Add deferred task resume flow to task lifecycle diagram
* #68139 - Add TaskGroup-level retries to retry a whole group as a unit
* #71108 - Dedent task docstrings used as doc_md
* #71549 - Fix task duration increasing forever after Dag run timeout skips it
* #71758 - Ignore task start_date and end_date on manual runs
* #71922 - Make HttpSensor defer when response_check is provided
* #72089 - Flag ui-field-behaviour drift between hooks and provider.yaml in static checks
* #72638 - Skip GCS folder-marker keys in GCSToSFTPOperator
* #73677 - i18n(ko): Add missing translations (Sep25)
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
* #69619 - Show task state counts for the latest Dag run on the Dags list page
* #70470 - UI: Add a time-window scope to the Dags list run state filter
This is **not** a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.
What we ask you to do is to make your **first prioritization decision**: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or `gh pr reopen <PR_NUMBER> --repo apache/airflow`. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.
While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.
Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.
---
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TaskGroup-level retries to retry a whole group as a unit [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #68139: Add TaskGroup-level retries to retry a whole group as a unit
URL: https://github.com/apache/airflow/pull/68139
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TaskGroup-level retries to retry a whole group as a unit [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #68139:
URL: https://github.com/apache/airflow/pull/68139#issuecomment-4798346943
@yuseok89 — I've removed the `ready for maintainer review` label because the next step here is yours: this branch now has **merge conflicts with `main`**. It'll go back into the maintainer queue automatically once you rebase onto the latest `main` and push — no need to re-add the label by hand. There's no rush.
<sub>Automated triage note drafted by an AI-assisted tool — may get things wrong; a real Apache Airflow maintainer takes the next look once it's green. ([why automated](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md))</sub>
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TaskGroup-level retries to retry a whole group as a unit [airflow]
Posted by "yuseok89 (via GitHub)" <gi...@apache.org>.
yuseok89 commented on code in PR #68139:
URL: https://github.com/apache/airflow/pull/68139#discussion_r3652772745
##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1198,6 +1201,61 @@ def _emit_dagrun_span(self, state: DagRunState):
span.set_status(status_code)
span.end()
+ def get_task_group_retry_count(self, group_id: str) -> int:
+ """Return how many times ``group_id`` has been retried as a unit in this run."""
+ return (self.task_group_retries or {}).get(group_id, 0)
+
+ def increment_task_group_retry_count(self, group_id: str) -> int:
+ """Bump and return the unit-retry counter for ``group_id`` in this run."""
+ counters = dict(self.task_group_retries or {})
+ counters[group_id] = counters.get(group_id, 0) + 1
+ self.task_group_retries = counters
+ return counters[group_id]
+
+ def _handle_task_group_retries(self, dag: SerializedDAG, *, session: Session) -> None:
+ """Clear and re-run any retryable TaskGroup that has a failed task and still has retries left."""
+ from airflow.models.taskinstance import clear_task_instances
+
+ retryable_groups = [
+ group for group in dag.task_group.get_task_group_dict().values() if group.retries > 0
+ ]
+ if not retryable_groups:
+ return
+
+ # Not-yet-started members are skipped, since clearing them records spurious history rows.
+ clearable_states = State.finished | {TaskInstanceState.RUNNING, TaskInstanceState.RESTARTING}
Review Comment:
I think I only had the sequential case in mind here.
You're right that it's a real gap.
Thanks for catching it, and fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TaskGroup-level retries to retry a whole group as a unit [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on code in PR #68139:
URL: https://github.com/apache/airflow/pull/68139#discussion_r3642864904
##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1198,6 +1201,61 @@ def _emit_dagrun_span(self, state: DagRunState):
span.set_status(status_code)
span.end()
+ def get_task_group_retry_count(self, group_id: str) -> int:
+ """Return how many times ``group_id`` has been retried as a unit in this run."""
+ return (self.task_group_retries or {}).get(group_id, 0)
+
+ def increment_task_group_retry_count(self, group_id: str) -> int:
+ """Bump and return the unit-retry counter for ``group_id`` in this run."""
+ counters = dict(self.task_group_retries or {})
+ counters[group_id] = counters.get(group_id, 0) + 1
+ self.task_group_retries = counters
+ return counters[group_id]
+
+ def _handle_task_group_retries(self, dag: SerializedDAG, *, session: Session) -> None:
+ """Clear and re-run any retryable TaskGroup that has a failed task and still has retries left."""
+ from airflow.models.taskinstance import clear_task_instances
+
+ retryable_groups = [
+ group for group in dag.task_group.get_task_group_dict().values() if group.retries > 0
+ ]
+ if not retryable_groups:
+ return
+
+ # Not-yet-started members are skipped, since clearing them records spurious history rows.
+ clearable_states = State.finished | {TaskInstanceState.RUNNING, TaskInstanceState.RESTARTING}
Review Comment:
Why does clearable_states only cover these states? What happens if a member is in `TaskInstanceState.DEFERRED` (or others) when the group retries?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org