[jira] [Commented] (AIRFLOW-433) Allow tasks to be scheduled in the current schedule interval instead of previous

["Daniel Imberman (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069136#comment-17069136 ] 

Daniel Imberman commented on AIRFLOW-433:
-----------------------------------------

closing as 5172 has this

> Allow tasks to be scheduled in the current schedule interval instead of previous
> --------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-433
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-433
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: Vineet Goel
>            Priority: Major
>
> I understand that airflow chose to schedule dag runs for the previous schedule interval because of the way data pipelines for batch processing are usually designed. For example, currently a daily job is scheduled for a dag run with execution date of Monday on a Tuesday. 
> However this is a bit weird for jobs that need to provide the same day of running as the execution_date (or {{ ds }}). I can get around this by using the date_add macro function for daily jobs but this workaround fails in the case of jobs that are scheduled to run on weekdays and should provide the same day in the context.
> It might be a good idea to make this configurable for a given dag as to whether we want the scheduler to schedule for the same interval or the previous. I understand by reading the code that this might be a bigger feature addition than it seems as this might involve major changes to how the scheduler works but this should be a good feature to have as airflow is a great tool to use for non batch processing jobs as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)