You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Lee-W (via GitHub)" <gi...@apache.org> on 2026/05/23 07:50:11 UTC
[I] Clarify partition_key semantics across DagRun / AssetEvent / runtime [airflow]
Lee-W opened a new issue, #67368:
URL: https://github.com/apache/airflow/issues/67368
### Body
## Context
In #66782 we wire up server-side consumption of task-emitted partition keys. The resulting `AssetEvent.partition_key` depends on whether the task touched
`outlet_events[asset]`:
| Task does... | `AssetEvent.partition_key` |
|---|---|
| nothing | inherits `DagRun.partition_key` |
| sets `extra` only, no `add_partitions()` | `None` |
| `add_partitions("us", "eu")` | `"us"`, `"eu"` |
Rows 1 and 2 look the same from the Dag author's side ("I didn't say anything about partitions") but produce different events. Before we lock this in for the 3.2 release, we should pin down what each field means.
## Proposed framing
- **`DagRun.partition_key` = provenance.** The partition this run was triggered on. Not a default for events the run produces.
- **`AssetEvent.partition_key` = routing pointer.** The key downstream partition mappers consume to decide which downstream DagRun to queue.
## What that implies
- Silent task → event inherits `DagRun.partition_key`. "I didn't say anything" is Identity passthrough of the run's own routing key.
- Row 2 is a bug under this framing — the producer was on a partition slice but left no routing pointer, so downstream partitioned consumers can never trigger. Normalise to row 1.
- `add_partitions("us", "eu")` is explicit fan-out — one event per routing pointer.
- `outlet_events[alias].add_partitions(...)` → no-op + warning. Aliases expand at queue time and have no routing of their own.
- Manual `partition_key` on a non-partitioned Dag → reject.
- Runtime (`PartitionAtRuntime`) silent task → event is `None`. No
provenance to inherit.
- Runtime back-fill of `DagRun.partition_key` from emitted keys is wrong under provenance framing — the run wasn't triggered on those keys, it discovered them. Drop it.
- n-1: downstream `DagRun.partition_key` (provenance, target space) and consumed `AssetEvent.partition_key` (routing, source space) are different by design. The mapping stays in `PartitionedAssetKeyLog`.
- 1-n: same routing pointer, different target per downstream via each downstream's mapper. Upstream read source off the event, downstream read target off the triggered DagRun.
## Today vs. proposal
| Rule | Today | Change if accepted |
|---|---|---|
| Silent task inherits run pk | `_register` fallback in `taskinstance.py` already does this | no change |
| Row 2 normalised | per-emission `partition_key=None` is preserved, event ends up `None` | in `_register`, when
`payload.partition_key is None`, substitute `dag_run_partition_key` |
| `add_partitions(...)` fan-out | works | no change |
| Alias `add_partitions` | silently dropped in `_serialize_outlet_events`; only the manager comment hints at it | warn (and
ignore) in `OutletEventAccessor.add_partitions` when key is `AssetAliasUniqueKey` |
| Manual pk on non-partitioned Dag | accepted, stored, ignored | reject at REST/CLI trigger layer based on `timetable.partitioned`
/ `partitioned_at_runtime` |
| Runtime silent task → `None` | depends on whether back-fill has fired first this turn | falls out for free once back-fill is
dropped |
| Runtime `DagRun.partition_key` back-fill | `taskinstance.py:1519-1525` back-fills when exactly 1 distinct emitted pk | drop the
block |
| n-1 / 1-n key spaces | implemented via `PartitionedAssetKeyLog` | doc-only — make the source/target distinction explicit in
`assets.rst` |
All of this is unreleased (3.2.0 ships AIP-76), so changes can land in
place — no compat shim needed.
Related: #59295, #59050, #67239, #66782.
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Clarify partition_key semantics across DagRun / AssetEvent / runtime [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W closed issue #67368: Clarify partition_key semantics across DagRun / AssetEvent / runtime
URL: https://github.com/apache/airflow/issues/67368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Clarify partition_key semantics across DagRun / AssetEvent / runtime [airflow]
Posted by "anishgirianish (via GitHub)" <gi...@apache.org>.
anishgirianish commented on issue #67368:
URL: https://github.com/apache/airflow/issues/67368#issuecomment-4585202921
Thanks so much for the PR, @Lee-W, and sorry for the late reply! 🙏 I went through the changes, and they look really good to me.
thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Clarify partition_key semantics across DagRun / AssetEvent / runtime [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #67368:
URL: https://github.com/apache/airflow/issues/67368#issuecomment-4535488911
> I completely agree on dropping the back-fill. Only thing I'd love to keep is the "this run handled partition X" view, we could pull that from the AssetEvents instead, maybe as a small follow-up.
Do you mean store this information in `DagRun.parition_key` or `AssetEvent.partition_key`? How would you like to express the idea that this run handles partition X, since it is decided at runtime and can have multiple values? Concatenation is probably one way, but not ideal 🤔
> On the manual-pk reject: I was poking around the trigger paths and got a little curious how you're picturing it landing. Looks like the public UI/REST trigger goes through create_dagrun directly while CLI/exec-API go via _trigger_dag, and create_dagrun is also where the scheduler/backfill set partition_key legitimately. Were you thinking one gated spot, or a check per path? Just want to make sure the UI trigger gets caught too.
IIRC, the UI simply won't show the column if it's not a partitioned Dag (still need to handle the REST side though). but yep, if we could have a one gated spot, that would be awesome. But if it took us too much time to refactor, we could do a check per path and refactor later
> +1 on the warn-and-ignore for alias add_partitions, and on rejecting for partitioned_at_runtime.
We could probably do this at some point, but let's get other things done before we go back to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Clarify partition_key semantics across DagRun / AssetEvent / runtime [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #67368:
URL: https://github.com/apache/airflow/issues/67368#issuecomment-4524686742
@anishgirianish took me some time to think about it. But this is what I'm thinking. let me know what do you thing 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Clarify partition_key semantics across DagRun / AssetEvent / runtime [airflow]
Posted by "anishgirianish (via GitHub)" <gi...@apache.org>.
anishgirianish commented on issue #67368:
URL: https://github.com/apache/airflow/issues/67368#issuecomment-4530013252
> [@anishgirianish](https://github.com/anishgirianish) took me some time to think about it. But this is what I'm thinking. let me know what do you thing 🙂
@Lee-W Thank you so much for writing this up. The provenance vs. routing split really makes sense to me.
Couple of small thoughts:
- I completely agree on dropping the back-fill. Only thing I'd love to keep is the "this run handled partition X" view — we could pull that from the AssetEvents instead, maybe as a small follow-up.
- On the manual-pk reject: I was poking around the trigger paths and got a little curious how you're picturing it landing. Looks like the public UI/REST trigger goes through create_dagrun directly while CLI/exec-API go via _trigger_dag, and create_dagrun is also where the scheduler/backfill set partition_key legitimately. Were you thinking one gated spot, or a check per path? Just want to make sure the UI trigger gets caught too.
- +1 on the warn-and-ignore for alias add_partitions, and on rejecting for partitioned_at_runtime.
I'd be glad to pick this up and work through it. Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org