[GitHub] [airflow] blcksrx opened a new issue, #25265: support continues catchup and backfill

[GitBox <gi...@apache.org>]

blcksrx opened a new issue, #25265:
URL: https://github.com/apache/airflow/issues/25265

   ### Description
   
   catchup and backfill provided good flexibility  to run `Dags` in pasts. the only thing that might concerning on those context is that they might produces a lot of `DagRun`. this situation could leads to some performance issues such as remaining slot availability and long time to finish.
   Assume that there is a DAG with `SqlOperator` using this sql template that runs every 15 mins:
   
   ```SQL
   SELECT * FROM table WHERE created_at BETWEEN {{  prev_data_interval_start_success }} AND {{ ts }}
   ```
   
   In case of any interruption on the scheduler level such as 1 day, this `catchup` process would creates 96 `DagRun`.
   or also assume running this `Dag` with backfill like this:
   ```BASH
   airflow backfill DAG --start-date=today --end-date=prev_week
   ```
   that would creates 672 `DagRun`!
   
   with simple feature that only runs 1 instance of `DagRun` and fulfils these parameters by the starts of the gap and end of the gap, this issue will be solve.
   
   ### Use case/motivation
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #25265: support continues catchup and backfill

[GitBox <gi...@apache.org>]

potiuk closed issue #25265: support continues catchup and backfill
URL: https://github.com/apache/airflow/issues/25265


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org