You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ashb (via GitHub)" <gi...@apache.org> on 2026/10/16 20:03:00 UTC
[PR] Resolve loop iterations through family and path columns on dynamic regions [airflow]
ashb opened a new pull request, #74892:
URL: https://github.com/apache/airflow/pull/74892
First off: why is this needed, wasn't loops landed and working? "Yes, but."
Part of the goal with the data-modelling changes for task loops (AIP-111)
is to support dynamic graphs (AIP-113) and other dynamic things, such as
choosing to allow loops inside loops (currently a parse error), with no future
migrations or data modelling changes needed. This change lets us enable these
features without expensive recursive queries in TIDep evaulation etc. And why
now? By getting this change in before 3.4.0 enters the RC stage, while we can
still change the migration "at zero-cost". And adding it as a migration in 3.5
would involve a (possibly costly) migration.
So the meat of the change:
Finding the task instances that sit inside one iteration of a loop currently
means finding every region of the loop's fork chain (a fork is when a loop is
cleared part way through and some iterations run a-fresh), plus everything
nested under that iteration, however deep. The previous lookup resolved those
region ids with two recursive CTEs and then passed them to the task_instance
query as literal lists. The nested list holds every region under the
iteration, whichever task is being read, so once something under an iteration
can fan out again (nested loops, or dynamic graphs inside a loop) its size
grows with the product of those widths. Every lookup hits this path: the
scheduler's trigger-rule check of a loop task repeats it on each pass until
the task is ready, and every XCom read from inside a loop pays it on the API
server. Measured with 5,000 regions under one iteration, a lookup that returns
one row took about 30 ms on PostgreSQL and MySQL, almost all of it building,
sending and parsing the list; with 50,000 it took about 550 ms.
Two columns on dynamic_region lets us get the same result but without
recursion. `family_root_id` is the id of the first region of a fork chain, so
a chain is one equality column comparisson. `path` contains, for each ancestor
from the top, its family root and the index the region sits at inside it, so
everything under one iteration of a loop shares a byte prefix and is one range
of an index. The lookup becomes a UNION ALL of two joins from task_instance
to dynamic_region, each driven by its own index; a single OR across the two
legs gives the planner no such choice. Its cost no longer depends on how many
regions sit under the iteration. The hook that fills the columns rejects a
fork whose parent differs from its source's, because both the family equality
and the prefix range rely on every member of a chain sharing one position.
Region inserts are rare next to task instance reads and writes, so the hook
and the two extra indexes cost comparitavely little.
The columns are NOT NULL so that raw inserts which skip the ORM fail loudly
instead of writing rows the lookup would silently miss. They are added to the
migration that creates dynamic_region, which has only shipped in a beta, so no
backfill is needed. A 3.4.0b1 database already ran that migration without them
so users in that situation will need to downgrade to 3.3 and upgrade again,
or reset the database. Nesting is capped at ten ancestors, which sets the
200-byte path width on MySQL. The cap is a chosen bound rather than an index
limit, and authoring already rejects nested loops, so current real depth is far lower.
In order to allow more easily resizing this column on MySQL in the future if
we want or need to, the length needs to be over 255 bytes (as InnoDB stores
VARBINARY with shorter values with a 1 byte length prefix.) So we have chosen
300, allowing 15 levels, such that we can change it in future with a meta-data
only storage change. Regardless of max length of column, all DBs only use as
much storage as needed (+ size counter).
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve loop iterations through family and path columns on dynamic regions [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #74892:
URL: https://github.com/apache/airflow/pull/74892#discussion_r4299785497
##########
airflow-core/src/airflow/migrations/versions/0143_3_4_0_add_dynamic_region_storage.py:
##########
@@ -157,6 +157,10 @@ def upgrade():
sa.Column("parent_region_id", CompactUUID(), nullable=True),
sa.Column("parent_region_index", sa.Integer(), nullable=True),
sa.Column("forked_from_region_id", CompactUUID(), nullable=True),
+ sa.Column("family_root_id", CompactUUID(), nullable=False),
Review Comment:
A 3.4.0b1 database is already stamped past this revision (b1 shipped 0143 to 0146), so `airflow db migrate` won't add these columns and nothing tells the user. The first sign is a missing `family_root_id` error from `DynamicRegion.get_or_create_many`, which `_create_tasks` calls for every mapped task, so a b1 user who never wrote a loop hits it too. Could we add a check to `_check_migration_errors` in `utils/db.py` that stops with a "migrated by 3.4.0b1, downgrade or reset" message when `dynamic_region` has no `family_root_id`? `upgradedb` already exits on those before running any migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve loop iterations through family and path columns on dynamic regions [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #74892:
URL: https://github.com/apache/airflow/pull/74892
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve loop iterations through family and path columns on dynamic regions [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #74892:
URL: https://github.com/apache/airflow/pull/74892#discussion_r4299785512
##########
airflow-core/src/airflow/models/dynamic_region.py:
##########
@@ -404,6 +446,70 @@ def _set_slot_key(mapper: Mapper, connection: Connection, region: DynamicRegion)
)
+def _get_coordinates_cache(region: DynamicRegion) -> dict[UUID, tuple[UUID, bytes]]:
+ """
+ Remember each region's coordinates until the transaction ends, so siblings cost one read of their parent.
Review Comment:
`after_transaction_end` also fires for the subtransaction every flush opens and for each `begin_nested()`, so this cache is cleared at the end of every flush, not the transaction. Behaviour is fine (a `get_or_create_many` batch shares one flush, and clearing on savepoint end keeps a rolled-back region out), so maybe just say "until the flush or savepoint ends" here. `test_coordinates_of_a_rolled_back_region_are_forgotten` and `test_coordinates_are_forgotten_when_the_transaction_commits` have the same mismatch: `add_region` has already flushed, so the cache is gone before the commit or rollback they're named for.
##########
airflow-core/src/airflow/migrations/versions/0143_3_4_0_add_dynamic_region_storage.py:
##########
@@ -157,6 +157,10 @@ def upgrade():
sa.Column("parent_region_id", CompactUUID(), nullable=True),
sa.Column("parent_region_index", sa.Integer(), nullable=True),
sa.Column("forked_from_region_id", CompactUUID(), nullable=True),
+ sa.Column("family_root_id", CompactUUID(), nullable=False),
Review Comment:
A 3.4.0b1 database is already stamped past this revision (b1 shipped 0143 to 0146), so `airflow db migrate` won't add these columns and nothing tells the user. The first sign is a missing `family_root_id` error from `DynamicRegion.get_or_create_many`, which `_create_tasks` calls for every mapped task, so a b1 user who never wrote a loop hits it too. Could we add a check to `_check_migration_errors` in `utils/db.py` that stops with a "migrated by 3.4.0b1, downgrade or reset" message when `dynamic_region` has no `family_root_id`? `upgradedb` already exits on those before running any migration.
##########
airflow-core/src/airflow/models/dynamic_region.py:
##########
@@ -457,59 +563,37 @@ def loop_position(
return None
-@attrs.define(frozen=True)
-class _LoopPassRegions:
- """The regions that hold one iteration of a loop: its fork family and everything nested under it."""
-
- family_ids: tuple[UUID, ...]
- nested_ids: tuple[UUID, ...]
- iteration: int
-
-
-def _load_loop_pass_regions(position: tuple[UUID, int], *, session: Session) -> _LoopPassRegions:
+def build_loop_family_legs(
+ query: Select[Any], family_member: DynamicRegion, *, iteration: int | None = None
+) -> tuple[Select[Any], Select[Any]]:
"""
- Resolve, in one statement, the regions that hold the loop iteration ``position`` names.
-
- Both recursive CTEs run against ``dynamic_region`` only. The ``task_instance`` query then takes the
- ids as literals, which every backend turns into an index lookup on ``task_instance_current_key``;
- a subquery in that position leaves MySQL and Postgres scanning every row of the task.
+ Split ``query``, a selection from ``task_instance``, into the rows that belong to one loop family.
+
+ ``family_member`` is any region of the loop's family; the family's regions share ``family_root_id`` and
+ ``path``. The first leg holds the rows that sit directly in a region of the family, where
+ ``region_index`` is the iteration. The second holds the rows nested under the family, however deep,
+ whose ``path`` starts with the family and the iteration. Pass ``iteration`` to keep one iteration, or
+ leave it out for all of them.
+
+ Each leg is joined to :class:`DynamicRegion`, so a caller can add columns to both (the iteration of
+ a nested row is ``DynamicRegion.parent_region_index``) before combining them with ``UNION ALL``. The
Review Comment:
For a row two or more levels under the family, the joined region's `parent_region_index` is its index inside the intermediate region, not the loop iteration. The `SUBSTRING(path, LENGTH(m.path) + 17, 4)` expression in the class docstring is the right one there. Worth saying "for a row directly under the family" or pointing at that instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve loop iterations through family and path columns on dynamic regions [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #74892:
URL: https://github.com/apache/airflow/pull/74892#issuecomment-6240751200
### Backport failed to create: v3-4-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/39067963861'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-4-test</td>
<td><a href="https://github.com/apache/airflow/commit/6a925a8fbe690444a8e685fdd824e79a5864833d"><img src='https://img.shields.io/badge/Commit-6a925a8-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker 6a925a8 v3-4-test
```
This should apply the commit to the v3-4-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve loop iterations through family and path columns on dynamic regions [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #74892:
URL: https://github.com/apache/airflow/pull/74892#issuecomment-6247562285
### Backport successfully created: v3-4-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-4-test</td>
<td><a href="https://github.com/apache/airflow/pull/74963"><img src="https://img.shields.io/badge/PR-74963-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org