[GitHub] [airflow] potiuk commented on issue #17010: Add support for custom Trigger Rule logic

[GitBox <gi...@apache.org>]

potiuk commented on issue #17010:
URL: https://github.com/apache/airflow/issues/17010#issuecomment-968385186


   To be honest it does not seem as bad when it comes to a "size" of change. 
   
   There are not that many places where trigger_rule is used (< 80 cases from a quick look) - so changing the rules to those rules would be possible.
   
   However I am not sure if that is needed. Maybe just the name of the rule should be changed (NONE_FAILED_OR_SKIPPED -> WAIT_FOR_ALL_AND_ONE_SUCCESS - it could could even be an alias.
   
   I think not everything that can be done, should be done. 
   
   If we allow for combine several triggers, there will be nonsense combinations (ALL_SUCCESS & ONE_FAILED for example) and we would have to detect all such cases when such combinations and mark them as invaild). That sounds really dangerous IMHO and opens a new class of potential erros when defining DAGS. I really like that people have predefined set of triggers that they can use rather than freely combine them. I think this is a bit too much freedom - with very little benefit, and a lot of potential problems.
   
   Very similarly - we could add (I actually had a working POC) MultiExecutor - to be able to combine any of the executors we have but eventually we only have CeleryKubernetesExecutor as "another executor" as this is the combination of executors that gives 90% of cases and saves us solving a lot of problems for people who would like to combine say - sequential executor. local and K8S one. 
   
   Additionally it opens up new cases which can add even more confusion and problems - like I think throwing calendar in the mix as a trigger is a really, really bad idea. Previously you only looked at the state of upstream tasks to determine whether the trigger fired. If we add logically connected calendar this is really changing the whole "logic" of triggers - and it is far too much IMHO. The "day" based triggerer is not "structural" decision, it is "time" decision and it should not be mixed in here..
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org