You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "FelipeArisi (via GitHub)" <gi...@apache.org> on 2024/10/30 16:33:13 UTC
[I] Add scheduler by Dataset and/or interval [airflow]
FelipeArisi opened a new issue, #43518:
URL: https://github.com/apache/airflow/issues/43518
### Description
In recent features it is possible to add dataset scheduling with conditional expressions (https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html#advanced-dataset-scheduling-with-conditional-expressions)
The idea would be to add a time-based schedule to these expressions as well. For example: schedule=['0 0 1 * *' | Dataset(path)]
### Use case/motivation
There are some events involving many Datasets as triggers. If a Dataset is late or gives an error, the entire pipeline will depend on it. The idea would be to add a time condition to start the DAG anyway.
Another problem is, if all the datasets are ready and the pipeline is ready, the pipeline must run after a certain time.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add scheduler by Dataset and/or interval [airflow]
Posted by "Saratqpy (via GitHub)" <gi...@apache.org>.
Saratqpy commented on issue #43518:
URL: https://github.com/apache/airflow/issues/43518#issuecomment-3148619559
Really interesting proposal — I’ve been working on a somewhat related scheduling case and wanted to see if this has come up elsewhere.
In my situation, the requirement is kind of the opposite: I need the DAG to trigger only when both a specific dataset has been updated AND a scheduled cron tick has occurred. Basically an AND condition instead of OR.
The reason is to avoid premature runs — for example, if the dataset is delayed or missing, we want to wait until the next cron window *and* make sure the data is ready.
I’m curious whether this kind of logic (cron AND dataset) is something that’s been discussed or considered for future versions?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add scheduler by Dataset and/or interval [airflow]
Posted by "FelipeArisi (via GitHub)" <gi...@apache.org>.
FelipeArisi closed issue #43518: Add scheduler by Dataset and/or interval
URL: https://github.com/apache/airflow/issues/43518
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add scheduler by Dataset and/or interval [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #43518:
URL: https://github.com/apache/airflow/issues/43518#issuecomment-2452146949
Does DatasetOrTimeSchedule help your use case?
https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timetable.html#dataset-event-based-scheduling-with-time-based-scheduling
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add scheduler by Dataset and/or interval [airflow]
Posted by "FelipeArisi (via GitHub)" <gi...@apache.org>.
FelipeArisi commented on issue #43518:
URL: https://github.com/apache/airflow/issues/43518#issuecomment-2456834444
I didn't know this one! It will work, thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org