[jira] [Commented] (AIRFLOW-4444) Parallelize handling of executor events

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-4444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16830560#comment-16830560 ] 

ASF GitHub Bot commented on AIRFLOW-4444:
-----------------------------------------

kunalyogenshah commented on pull request #5215: [AIRFLOW-4444][WIP] Introducing ghosts and parallel executor event handling
URL: https://github.com/apache/airflow/pull/5215
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW-4444)
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI changes:
   
   This PR is currently a WIP, but establishes the core changes that are being proposed.
    - Executor events are being re-termed as ghosts (to keep in line with the zombies nomenclature). Ghosts are events that are recorded as QUEUED in the DB, but executor has marked them as finished.
    - Ghosts are stateful if discovered using the event buffer, and hence need to be handled sequentially and aggressively. Ghosts can be stateless if discovered using the DB first, and then mapped against executor directly.
    - If the discovery is stateless, then we can defer the handling of ghosts, and push them just like zombies to the `DagFileProcessor` instances for handling.
   
   Main changes :
    - Ghosts will be discovered in a single query in a scheduler heartbeat. (previously it was a single query per ghost)
    - Ghosts will be lazily handled when the DAG is eventually parsed again by the processors.
    - Ghosts won't appear individually inside the logs as executor events, they will appear as zombies instead.
    - Each DAG is guaranteed to be parsed only once per loop.
   
   This PR addresses an important concern for us as we see Scheduler loops sometimes exceed 100s if there is a big accumulation of executor events.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   
   TODO, will add next.
   
   ### Commits
   
   ### Documentation
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Parallelize handling of executor events
> ---------------------------------------
>
>                 Key: AIRFLOW-4444
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4444
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Kunal Shah
>            Assignee: Kunal Shah
>            Priority: Major
>
> The _process_executor_events function today performs sequential reading of DAGs when it encounters an event like : "TI is queued but executor has marked TI as succeeded or failed". Each such TI is individually parsed in sequence and then handled for failure.
> This Jira proposes calling these events ghosts (like zombies, but different), make their discovery stateless, and handle them through the same framework created for zombies to provide parallelization. This will be a departure from existing logic.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)