You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "liyakun (JIRA)" <ji...@apache.org> on 2019/07/02 07:37:00 UTC
[jira] [Assigned] (YARN-9480) createAppDir() in
LogAggregationService shouldn't block dispatcher thread of
ContainerManagerImpl
[ https://issues.apache.org/jira/browse/YARN-9480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liyakun reassigned YARN-9480:
-----------------------------
Assignee: Yunyao Zhang (was: liyakun)
> createAppDir() in LogAggregationService shouldn't block dispatcher thread of ContainerManagerImpl
> -------------------------------------------------------------------------------------------------
>
> Key: YARN-9480
> URL: https://issues.apache.org/jira/browse/YARN-9480
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Reporter: liyakun
> Assignee: Yunyao Zhang
> Priority: Major
>
> At present, when startContainers(), if NM does not contain the application, it will enter the step of INIT_APPLICATION. In the application init step, createAppDir() will be executed, and it is a blocking operation.
> createAppDir() is an operation that needs to interact with an external file system. This operation is affected by the SLA of the external file system. Once the external file system has a high latency, the NM dispatcher thread of ContainerManagerImpl will be stuck. (In fact, I have seen a scene that NM stuck here for more than an hour.)
> I think it would be more reasonable to move createAppDir() to the actual time of uploading log (in other threads). And according to the logRetentionPolicy, many of the containers may not get to this step, which will save a lot of interactions with external file system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org