[GitHub] [airflow] potiuk commented on issue #28382: Implement all_done_min_one_success trigger rule

[GitBox <gi...@apache.org>]

potiuk commented on issue #28382:
URL: https://github.com/apache/airflow/issues/28382#issuecomment-1353775457

   Is there any reason you can't have an "all_done" task that checks if there is an upstream task that succeeded and trigger data loading then?
   
   ```
   INGEST 1. ----\
   
   INGEST 2  ----- (all_done) --> CHECK ANY UPSTREAM SUCCEEDED [fail if there none of the upstreams succeeded] -> LOAD 
   
   INGEST 3  ----/
   ```
   
   IMHO, it is fully doable, does not require to add any complexity and adding more complex trigger rules, it very well reflects your use case (you anyhow need to wait with the decision until all of the ingestion tasks are done), and it also gives you clear  indication of the case where the "CHECK ALL UPSTREAMS" task will clearly inform you that this was there reasons. 
   
   Checking the state of those tasks can be easily done by XCom (and if you do the "CHECK" task as @task decorated operator, it is as simple as checking the output of all of the tasks. Moreover the "check" task could simply pass the list of all ingested datasets as a single input tho the LOAD task, so that it does not have to rely on checking if any of the tasks failed or not.
   
   For me this is a very nice separation of concerns - having one task to decide whether to proceed or not based on deep analysis of which of those tasks succeed and which did not, and separately having a LOAD tasks that loads whatever you pass to it - without worrying about states there. I am afraid your LOAD task in your case will have to that kind of check anyway if you would like to get the rule you want (but in a multi-function task). Splitting those seems much cleaner (and you could make yur "CHECK" task reusable across several different LOAD tasks as well - without having to implement the same "select only those that succeeded" logic.
   
   Anything wrong with this approach in your case?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org