[jira] [Commented] (AIRFLOW-4444) Parallelize handling of executor events

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-4444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927666#comment-16927666 ] 

ASF GitHub Bot commented on AIRFLOW-4444:
-----------------------------------------

stale[bot] commented on pull request #5215: [AIRFLOW-4444][WIP] Introducing ghosts and parallel executor event handling
URL: https://github.com/apache/airflow/pull/5215
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Parallelize handling of executor events
> ---------------------------------------
>
>                 Key: AIRFLOW-4444
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4444
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executors
>            Reporter: Kunal Shah
>            Assignee: Kunal Shah
>            Priority: Major
>
> The _process_executor_events function today performs sequential reading of DAGs when it encounters an event like : "TI is queued but executor has marked TI as succeeded or failed". Each such TI is individually parsed in sequence and then handled for failure.
> This Jira proposes calling these events ghosts (like zombies, but different), make their discovery stateless, and handle them through the same framework created for zombies to provide parallelization. This will be a departure from existing logic.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)