[GitHub] [airflow] SamWheating commented on a diff in pull request #29909: Adding ContinuousTimetable and support for @continuous schedule_interval

["SamWheating (via GitHub)" <gi...@apache.org>]

SamWheating commented on code in PR #29909:
URL: https://github.com/apache/airflow/pull/29909#discussion_r1125601220


##########
airflow/timetables/simple.py:
##########
@@ -108,6 +109,37 @@ def next_dagrun_info(
         return DagRunInfo.exact(run_after)
 
 
+class ContinuousTimetable(_TrivialTimetable):
+    """Timetable that schedules continually, while still respecting start_date and end_date
+
+    This corresponds to ``schedule="@continuous"``.
+    """
+
+    description: str = "As frequently as possible while still obeying max_active_runs"

Review Comment:
   > Should we enforce "max_active_runs=1" for @continuous run ? If I understand correctly this is the only reasonable setting for @continuous, because if max_active_runs is not set or set to high number, scheduler will start scheduling and runnning a lot of dagruns for such dag.
   
   I thought about this but ultimately thought it would be weird to impose an artificial limitation like this. I think in some cases a user might want to have multiple runs executing at all times (for example, a job with multiple stages which uses `depends_on_past` for continuous pipelined execution). Additionally, a similar hazard already exists with `schedule_interval="* * * * *"` which could create many jobs quite quickly.
   
   So I would advocate for leaving this as-is and continuing to use `max_active_runs` as the limit, we can include some warnings about this potential hazard in the docs. Thoughts?
   
   > Besides this change also needs documentation and examples
   
   Yup, I'll get to work on those shortly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org