You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/08 06:30:31 UTC

[GitHub] [airflow] nullhack opened a new issue #9176: [Feature request] Add reverse run of DAGs

nullhack opened a new issue #9176:
URL: https://github.com/apache/airflow/issues/9176


   **Description of the problem**
   
   A common workflow we have is running hourly DAGs not dependent of past runs. When deployed, It needs to catch up for many years before the runs are up to date and can be used by analysts.
   
   The problem is that most of time, Recent runs are more important than performing ETL of execution time 20 years ago. In practice we deploy the DAG two times, one for filling recent data and one for filling since inception.
   
   **Feature requested**
   
   A flag ('back_run=False') on DAG class. If this flag is set to `True`, DAG runs are scheduled in reverse order.
   
   E.g. consider we're on day `10` and in the middle of the process a new run for day `11` happens:
   the DAG runs would be scheduled like `10 -> 9 -> 8 -> 11 -> 7 -> 6 -> 5` (assuming the run for `11` happened after running for day 8). This flag can only be used if `catchup=False`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] swapniel99 edited a comment on issue #9176: [Feature request] Add reverse run of DAGs

Posted by GitBox <gi...@apache.org>.
swapniel99 edited a comment on issue #9176:
URL: https://github.com/apache/airflow/issues/9176#issuecomment-641485098


   I think `back_run=True` can work only if `catchup=True` but `depends_on_past=False`
   `catchup=False` means past runs are not necessary.
   Can anyone correct me if I am wrong?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] swapniel99 commented on issue #9176: [Feature request] Add reverse run of DAGs

Posted by GitBox <gi...@apache.org>.
swapniel99 commented on issue #9176:
URL: https://github.com/apache/airflow/issues/9176#issuecomment-641485098


   I think `back_run=True` can work only if `catchup=True` but `depends_on_past=False`
   catchup=False means past runs are not necessary.
   Can anyone correct me if I am wrong?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] nullhack edited a comment on issue #9176: [Feature request] Add reverse run of DAGs

Posted by GitBox <gi...@apache.org>.
nullhack edited a comment on issue #9176:
URL: https://github.com/apache/airflow/issues/9176#issuecomment-641536838


   > I think `back_run=True` can work only if `catchup=True` but `depends_on_past=False`
   > `catchup=False` means past runs are not necessary.
   > Can anyone correct me if I am wrong?
   
   Yes, this is what I meant, thanks for spotting that. I'm changing the description


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] nullhack commented on issue #9176: [Feature request] Add reverse run of DAGs

Posted by GitBox <gi...@apache.org>.
nullhack commented on issue #9176:
URL: https://github.com/apache/airflow/issues/9176#issuecomment-641536838


   > I think `back_run=True` can work only if `catchup=True` but `depends_on_past=False`
   > `catchup=False` means past runs are not necessary.
   > Can anyone correct me if I am wrong?
   
   Yes, this is what I meant, thanks for spotting that. I'm changing the question


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org