You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nimrodV81 (via GitHub)" <gi...@apache.org> on 2024/04/18 08:10:56 UTC
[I] add option to set DAG level Dataset outlets [airflow]
nimrodV81 opened a new issue, #39105:
URL: https://github.com/apache/airflow/issues/39105
### Description
add a new argument `outlets` to the DAG constructor with the same effect as the BaseOperator's outlets argument when the full dag run is successfull
### Use case/motivation
use case:
I'm currently maintaining several DAGs with "Dataset Scheduling", where each DAG handles a specific dataset and declares that it is ready for downstream DAGs by an EmptyOperator with an `outlets` value
this means I need to fan-in all the leafs of each DAG:
suggestion:
add an `outlets` argument to the `DAG` constructor to mark the Dataset(s) as ready when the dag run is successful
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add option to set DAG level Dataset outlets [airflow]
Posted by "nimrodV81 (via GitHub)" <gi...@apache.org>.
nimrodV81 commented on issue #39105:
URL: https://github.com/apache/airflow/issues/39105#issuecomment-2074396575
@RNHTTR Thanks for the feedback
I want to add this feature myself, but I'm afraid I need some guidance
I've forked the repo and started to dive into the code
Can you point me to a good place to place on_success code for a `DagRun`
(similar to `airflow.models.taskinstance._run_finished_callback` for `TaskInstance`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add option to set DAG level Dataset outlets [airflow]
Posted by "ralichkov (via GitHub)" <gi...@apache.org>.
ralichkov commented on issue #39105:
URL: https://github.com/apache/airflow/issues/39105#issuecomment-3191003613
Hi @nimrodV81 did you get around implementing this?
Feature makes a lot of sense and I'm currently going to implement the same workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add option to set DAG level Dataset outlets [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #39105:
URL: https://github.com/apache/airflow/issues/39105#issuecomment-3986488040
@nimrodV81 We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add option to set DAG level Dataset outlets [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #39105:
URL: https://github.com/apache/airflow/issues/39105#issuecomment-2065314858
Initially I thought this was a perfect use case for `EmptyOperator`, but that's just one more thing to maintain in potentially many DAGs.
I've assigned you to this issue :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add option to set DAG level Dataset outlets [airflow]
Posted by "nimrodV81 (via GitHub)" <gi...@apache.org>.
nimrodV81 commented on issue #39105:
URL: https://github.com/apache/airflow/issues/39105#issuecomment-2173566009
@RNHTTR after some observation, the design I came up with is to create a callback and append it to `DAG.on_success_callback` during init
is this an acceptable solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add option to set DAG level Dataset outlets [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #39105:
URL: https://github.com/apache/airflow/issues/39105#issuecomment-2173618621
> @RNHTTR after some observation, the design I came up with is to create a callback and append it to `DAG.on_success_callback` during init is this an acceptable solution?
Interesting! Does this make an API call to trigger a Dataset and thus avoids any changes to the Airflow source code? Or does this take a new parameter in the DAG's initialization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add option to set DAG level Dataset outlets [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #39105:
URL: https://github.com/apache/airflow/issues/39105#issuecomment-2063283155
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org