[jira] [Created] (YARN-11407) short-live job log aggregation blocked on LRA

["Shuyi Zhang (Jira)" <ji...@apache.org>]

Shuyi Zhang created YARN-11407:
----------------------------------

             Summary: short-live job log aggregation blocked on LRA 
                 Key: YARN-11407
                 URL: https://issues.apache.org/jira/browse/YARN-11407
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: Shuyi Zhang


Currently, log aggregation threads are scheduled based on applications start time. This means later applications that start later but finish earlier, will be blocked on log aggregation, until the previous running jobs finish and release a log aggregation thread for it to use. To resolve this bottleneck, we can decouple the initialization from the thread scheduling of the ApplogAggregator:

Initializing an AppLogAggregator on application start
But scheduling it in the thread pool on application finish
This schedules log aggregation threads based on finish time instead application start time, so prevents having a pileup of jobs waiting for log aggregation, especially when waiting for long-running jobs to finish.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org