[GitHub] [airflow] noamelf commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run

[GitBox <gi...@apache.org>]

noamelf commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
URL: https://github.com/apache/airflow/pull/4751#discussion_r347107741
 
 

 ##########
 File path: airflow/ti_deps/deps/trigger_rule_dep.py
 ##########
 @@ -34,6 +35,31 @@ class TriggerRuleDep(BaseTIDep):
     IGNOREABLE = True
     IS_TASK_DEP = True
 
+    @staticmethod
+    def _get_states_count_upstream_ti(ti, finished_tasks):
+        """
+        :param ti the ti that we want to calculate deps for
+        :type ti airflow.models.TaskInstance
+        :param finished_tasks all the finished tasks of the dag_run
+        :type finished_tasks of finished ti's
+        """
+        successes, skipped, failed, upstream_failed, done = 0, 0, 0, 0, 0
+        upstream_tasks = [finished_task for finished_task in finished_tasks
+                          if finished_task.task_id in ti.task.upstream_task_ids]
+        if upstream_tasks:
+            upstream_tasks_sorted = sorted(upstream_tasks, key=lambda x: x.state)
+            for k, g in groupby(upstream_tasks_sorted, key=lambda x: x.state):
+                if k == State.SUCCESS:
+                    successes = len(list(g))
+                elif k == State.SKIPPED:
+                    skipped = len(list(g))
+                elif k == State.FAILED:
+                    failed = len(list(g))
+                elif k == State.UPSTREAM_FAILED:
+                    upstream_failed = len(list(g))
 
 Review comment:
   I think that by using a counter you can get the same results with simpler code:
   
   ```python
   Counter(task.state for task in upstream_tasks_sorted)
   ```
   
   And then organize the results this way:
   
   ```python
   counter[State.SKIPPED], counter[State.UPSTREAM_FAILED] # ...
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services