Re: [I] DAGs with future `start_date` are automatically marked as success without executing tasks [airflow]

["adamliter (via GitHub)" <gi...@apache.org>]

adamliter commented on issue #38513:
URL: https://github.com/apache/airflow/issues/38513#issuecomment-2513324931

   This is the same for manually-triggered DAG runs with a logical date that is before the `start_date`. I would expect/hope that manually triggered runs with a logical date prior to the `start_date` of the DAG would actually run. However, they do not (i.e.,  none of the tasks run; instead, the DAG run is just marked as an immediate success).
   
   I have a use case for this where I have a DAG that schedules a compute-intensive Spark job on an EMR cluster. Because of the nature of the upstream dataset (there is late-arriving data), the Spark job processes data with a 7-day lookback window from each `data_interval_start`. The job runs twice daily. As a result, some of the same data is reprocessed. I set the start date to `2024-12-01`. But I would like to backfill data from `2024-10-01` to `2024-12-01`.
   
   I don't want to set the `start_date` to `2024-10-01` with `catchup=True` because the late-arriving data has already arrived for the historical months. In other words, I don't want to have ≈120 DAG runs get scheduled for the preceding two months. Instead, I'd like to just run some manual DAG runs: `2024-10-07`, `2024-10-14`, `2024-10-21`, `2024-10-28`, _etc._. Since the late-arriving data has already arrived for these historical months, I don't want to have several overlapping runs with a sliding lookback window. Instead, I just want to trigger several manual runs each of which is spaced 7 days apart for the historical months.
   
   I see that @artilexx had done some initial work to address this in #40711 but it was closed due to inactivity.
   
   Is there still interest in merging this? If so, happy to take a shot at it when I can find time and also try to make it work for my use case with manually triggered DAG runs with a logical date prior to the `start_date`. (But also don't want to step on @artilexx's toes if they want to pick it back up.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org