[D] extra flag to make catchup=False mean "first run is next scheduled" [airflow]

["seth-olsen (via GitHub)" <gi...@apache.org>]

GitHub user seth-olsen created a discussion: extra flag to make catchup=False mean "first run is next scheduled"

### Description

Currently, `catchup=False` means "first run after turning on will be the latest scheduled run" and there is no way to produce the behavior where turning on a DAG means "make the first run after turning on occur at the next schedule time instead of the previous one".

Proposal: make a new flag (call it something to the effect of `no_catchup_means_no_past_runs`) which when set to `True` means that turning on a DAG that has the setting `catchup=False` will result in the first run being the one on the schedule that occurs next in time (rather than the one that occurs in the most recent past).



### Use case/motivation

The use case is whenever you want to turn on a DAG without it running right away.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)


GitHub link: https://github.com/apache/airflow/discussions/45777

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] extra flag to make catchup=False mean "first run is next scheduled" [airflow]

["boring-cyborg[bot] (via GitHub)" <gi...@apache.org>]

GitHub user boring-cyborg[bot] added a comment to the discussion: extra flag to make catchup=False mean "first run is next scheduled"

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.


GitHub link: https://github.com/apache/airflow/discussions/45777#discussioncomment-11874374

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] extra flag to make catchup=False mean "first run is next scheduled" [airflow]

["MaluNoPeleke (via GitHub)" <gi...@apache.org>]

GitHub user MaluNoPeleke added a comment to the discussion: extra flag to make catchup=False mean "first run is next scheduled"

Is there already a solution for this problem available?

GitHub link: https://github.com/apache/airflow/discussions/45777#discussioncomment-15027846

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] extra flag to make catchup=False mean "first run is next scheduled" [airflow]

["seth-olsen (via GitHub)" <gi...@apache.org>]

GitHub user seth-olsen added a comment to the discussion: extra flag to make catchup=False mean "first run is next scheduled"

As the thread and stale/closed PRs mention, there are many cases where the workaround with start_date is undesirable or even impossible. I'm not sure why neither of those PRs was accepted, seemed like a lot of the opposition was along the lines of "eh, I don't want to have to look at another argument in the config". Very glad I didn't waste time writing up a PR!

GitHub link: https://github.com/apache/airflow/discussions/45777#discussioncomment-11874376

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] extra flag to make catchup=False mean "first run is next scheduled" [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

GitHub user potiuk added a comment to the discussion: extra flag to make catchup=False mean "first run is next scheduled"

For now - I convert it to a discussion as this certainly not an issue or a simple feature that is clear whether or if we should follow it.

GitHub link: https://github.com/apache/airflow/discussions/45777#discussioncomment-11874378

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] extra flag to make catchup=False mean "first run is next scheduled" [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

GitHub user potiuk added a comment to the discussion: extra flag to make catchup=False mean "first run is next scheduled"

> "eh, I don't want to have to look at another argument in the config"

Yes. that's a very good reason in fact. Adding more confusion and options is not desireable "product" property. Sometimes even at the expense not handling all cases. You can have a product with million configurable parameters that is useless and far too generic. So "I do not want to have yet another knob to turn" is quite a good reason for not accepting it - from product point of view, even if individual cases are not happy. Generally it's impossible to make everyone happy, some people will still be somewhat unhappy.

This looks like an important change in behaviour that also might impact some of the other discussions we have about Airlfow 3 - namely a lot of discussions about https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-83+amendment+to+support+classic+Airflow+authoring+style and resulting in https://cwiki.apache.org/confluence/display/AIRFLOW/Option+2+clarification+doc+WIP . While not 100% related, this PR and the backfill change mentioned by Elad are very much related to the catchup behaviour and show how you should approach such discussions.

My suggestion is @seth-olsen if you feel very strongly about this one, start a discussion on devlist and put forth your arguments. Analyse all the past dicussions that @eladkal so helpfully provided, read them in detail, anylyse why thigns were rejected - try to understand other's arguments (even if you do not agree with them, trying to understand what others are saying is a good idea), and come up with a concrete proposal how you think your case should be addressed and justify it. Eventually everything we do is consensus driven (i.e. we want to get to consensus where generally we agree to a direction) but if we cannot reach consensus, the last resort is voting https://www.apache.org/foundation/voting.html

Note that - similarly to what Daniel did, when presenting your proposal you shoudl consider all the cases and combinations in your proposal, what it means what consequences it has when introduced, what it means for backwards compatibility etc.. Just thorough thinking followed by discussion, reaching consensus and if not possible, defining the outcome and calling for a vote. 

Your case is way simpler than what Daniel discussed, but the mechanism is very similar.

So I propose you start a `[DISCUSS]` thread on our devlist, where you explain rationale and refer to the past dicussions - but make sure it covers all cases and earlier arguments , let it run for a while and have people discuss it, drive it to consensus if possible and then call for a vote when you think general direction is set.


GitHub link: https://github.com/apache/airflow/discussions/45777#discussioncomment-11874377

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] extra flag to make catchup=False mean "first run is next scheduled" [airflow]

["eladkal (via GitHub)" <gi...@apache.org>]

GitHub user eladkal added a comment to the discussion: extra flag to make catchup=False mean "first run is next scheduled"

Related: https://lists.apache.org/thread/j2n8h308ffq46sx0vcfnl61snh7tyjlo
and https://github.com/apache/airflow/pull/38168, https://github.com/apache/airflow/pull/35392

The current behavior is consistent with how data pipeline works.
Today you are processing yesterday data. Thus, if you set catchup=False and you'd like to avoid creating the first run then your `start_date` needs to be adjusted with +1 day.

Generally speaking, this behavior is not so suitable for other patterns. The backfill pain will be resolved with [AIP-78 Scheduler-managed backfill](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-78+Scheduler-managed+backfill) the rest will be handled in Airflow 3.1+

GitHub link: https://github.com/apache/airflow/discussions/45777#discussioncomment-11874375

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org