You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kunal Shah (JIRA)" <ji...@apache.org> on 2019/04/30 18:22:00 UTC
[jira] [Created] (AIRFLOW-4444) Parallelize handling of executor
events
Kunal Shah created AIRFLOW-4444:
-----------------------------------
Summary: Parallelize handling of executor events
Key: AIRFLOW-4444
URL: https://issues.apache.org/jira/browse/AIRFLOW-4444
Project: Apache Airflow
Issue Type: Improvement
Reporter: Kunal Shah
Assignee: Kunal Shah
The _process_executor_events function today performs sequential reading of DAGs when it encounters an event like : "TI is queued but executor has marked TI as succeeded or failed". Each such TI is individually parsed in sequence and then handled for failure.
This Jira proposes calling these events ghosts (like zombies, but different), make their discovery stateless, and handle them through the same framework created for zombies to provide parallelization. This will be a departure from existing logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)