You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/28 14:43:59 UTC
[GitHub] [airflow] collinmcnulty opened a new issue, #26751: Mapped tasks each update a different Dataset
collinmcnulty opened a new issue, #26751:
URL: https://github.com/apache/airflow/issues/26751
### Description
It would be great is each map_index in a set of dynamically mapped tasks could each update a different Dataset depending on the parameters sent to them.
### Use case/motivation
Perform the same transformation on a number of similar datasets and then have a bunch of separate DAGs that do further analysis after each mapped task is finished. It seems relatively common to use mapped tasks to do the same thing to multiple datasets, so not being able to be aware of multiple datasets across a set of mapped tasks is something of a handicap.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Mapped tasks each update a different Dataset [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr closed issue #26751: Mapped tasks each update a different Dataset
URL: https://github.com/apache/airflow/issues/26751
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Mapped tasks each update a different Dataset [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #26751:
URL: https://github.com/apache/airflow/issues/26751#issuecomment-3489637598
It’s not directly supported, but AssetAlias can kind of do this with some custom code
```python
alias = AssetAlias(name="foo")
def look_up_asset(map_index) -> Asset:
...
@task(outlet=[alias])
def do(map_index, outlet_events):
outlet_events[alias].add(look_up_asset(map_index))
```
More on this in documentation: https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/assets.html#dynamic-data-events-emitting-and-asset-creation-through-assetalias
Does this sound good enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org