You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kunal Shah (JIRA)" <ji...@apache.org> on 2019/04/30 18:22:00 UTC

[jira] [Created] (AIRFLOW-4444) Parallelize handling of executor events

Kunal Shah created AIRFLOW-4444:
-----------------------------------

             Summary: Parallelize handling of executor events
                 Key: AIRFLOW-4444
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4444
             Project: Apache Airflow
          Issue Type: Improvement
            Reporter: Kunal Shah
            Assignee: Kunal Shah


The _process_executor_events function today performs sequential reading of DAGs when it encounters an event like : "TI is queued but executor has marked TI as succeeded or failed". Each such TI is individually parsed in sequence and then handled for failure.

This Jira proposes calling these events ghosts (like zombies, but different), make their discovery stateless, and handle them through the same framework created for zombies to provide parallelization. This will be a departure from existing logic.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)