[GitHub] [airflow] potiuk commented on pull request #29909: Adding ContinuousTimetable and support for @continuous schedule_interval

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk commented on PR #29909:
URL: https://github.com/apache/airflow/pull/29909#issuecomment-1455026267

   > I’m not sure a new directive is worthwhile tbh.
   
   I think it's an interesting pattern - a number of users asked for it and any attempt to do it without such directive is pretty cumbersome. While I initially had the same thought, Looking at how simple it will be for the users to use it, I think it is worthwile to add it. 
   
   I generally also think we should have more built-in time-tables that serve various cases like that. The custom timatable interface is prohibitively complex evn for an experienced Python developer and testing it is next to impossible. A good example of that is CronTriggerTimetable - which even is written by us, apparently exposes race condition: https://github.com/apache/airflow/issues/27399 where it occasionally looses one tasks. 
   
   So as a community I think we should invest IMHO in having a few more "generic", "robust" and "declaratively configurable" timetables that should serve a number of common cases as otherwise we are asking our users for too much of an effort to develop their own custom timetables. 
   
   This is just one example of such case and imho more cases should follow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org