You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nimrodV81 (via GitHub)" <gi...@apache.org> on 2024/04/18 08:10:56 UTC
[I] add option to set DAG level Dataset outlets [airflow]
nimrodV81 opened a new issue, #39105:
URL: https://github.com/apache/airflow/issues/39105
### Description
add a new argument `outlets` to the DAG constructor with the same effect as the BaseOperator's outlets argument when the full dag run is successfull
### Use case/motivation
use case:
I'm currently maintaining several DAGs with "Dataset Scheduling", where each DAG handles a specific dataset and declares that it is ready for downstream DAGs by an EmptyOperator with an `outlets` value
this means I need to fan-in all the leafs of each DAG:
suggestion:
add an `outlets` argument to the `DAG` constructor to mark the Dataset(s) as ready when the dag run is successful
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add option to set DAG level Dataset outlets [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #39105:
URL: https://github.com/apache/airflow/issues/39105#issuecomment-2065314858
Initially I thought this was a perfect use case for `EmptyOperator`, but that's just one more thing to maintain in potentially many DAGs.
I've assigned you to this issue :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add option to set DAG level Dataset outlets [airflow]
Posted by "nimrodV81 (via GitHub)" <gi...@apache.org>.
nimrodV81 commented on issue #39105:
URL: https://github.com/apache/airflow/issues/39105#issuecomment-2074396575
@RNHTTR Thanks for the feedback
I want to add this feature myself, but I'm afraid I need some guidance
I've forked the repo and started to dive into the code
Can you point me to a good place to place on_success code for a `DagRun`
(similar to `airflow.models.taskinstance._run_finished_callback` for `TaskInstance`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add option to set DAG level Dataset outlets [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #39105:
URL: https://github.com/apache/airflow/issues/39105#issuecomment-2063283155
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org