You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/01/01 00:29:00 UTC
[jira] [Commented] (YARN-11644) LogAggregationService can't upload log in time when application finished
[ https://issues.apache.org/jira/browse/YARN-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18048615#comment-18048615 ]
ASF GitHub Bot commented on YARN-11644:
---------------------------------------
github-actions[bot] commented on PR #6452:
URL: https://github.com/apache/hadoop/pull/6452#issuecomment-3703100084
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.
> LogAggregationService can't upload log in time when application finished
> ------------------------------------------------------------------------
>
> Key: YARN-11644
> URL: https://issues.apache.org/jira/browse/YARN-11644
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: log-aggregation
> Affects Versions: 3.3.6
> Reporter: Xie YiFan
> Assignee: Xie YiFan
> Priority: Minor
> Labels: pull-request-available
> Attachments: image-2024-01-10-11-03-57-553.png
>
>
> LogAggregationService is responsible for uploading log to HDFS. It applies thread pool to execute upload task.
> The workflow of upload log as follow:
> # NM construct Applicaiton object when first container of a certain application launch, then notify LogAggregationService to init AppLogAggregationImpl.
> # LogAggregationService submit AppLogAggregationImpl to task queue
> # The idle worker of thread pool pulls AppLogAggregationImpl from task queue.
> # AppLogAggregationImpl do while loop to check the application state, do upload when application finished.
> Suppose the following scenario:
> * LogAggregationService initialize thread pool with 4 threads.
> * 4 long running applications start on this NM, so all threads are occupied by aggregator.
> * The next short application starts on this NM and quickly finish, but no idle thread for this app to upload log.
> as a result, the following applications have to wait the previous applications finish before uploading their logs.
> !image-2024-01-10-11-03-57-553.png|width=599,height=195!
> h4. Solution
> Change the spin behavior of AppLogAggregationImpl. If application has not finished, just return to yield current thread and resubmit itself to executor service. So the LogAggregationService can roll the task queue and the logs of finished application can be uploaded immediately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org