You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2017/08/23 20:29:01 UTC
[jira] [Commented] (YARN-7087) NM failed to perform log aggregation
due to absent container
[ https://issues.apache.org/jira/browse/YARN-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139030#comment-16139030 ]
Jason Lowe commented on YARN-7087:
----------------------------------
Looks like this is related to YARN-221 and YARN-4152. The latter fixed the NPE issue introduced by the first, but unfortunately simply ignoring container IDs that are absent isn't a real fix. The end result when that scenario occurs is that we will always skip aggregating that container's logs, and that may or may not be the desired effect. In this case it was not.
I believe the scenario occurs because LogAggregationService has not seen the event requesting log aggregation before the NM heartbeats to the RM and then decides to remove the container because the app has completed. The aggregation service appears to only need to lookup the container to get the container type, so maybe we can simply store the container type in the log aggregation event so it doesn't need to lookup the container to process the event.
> NM failed to perform log aggregation due to absent container
> ------------------------------------------------------------
>
> Key: YARN-7087
> URL: https://issues.apache.org/jira/browse/YARN-7087
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
> Affects Versions: 2.8.1
> Reporter: Jason Lowe
> Priority: Critical
>
> Saw a case where the NM failed to aggregate the logs for a container because it claimed it was absent:
> {noformat}
> 2017-08-23 18:35:38,283 [AsyncDispatcher event handler] WARN logaggregation.LogAggregationService: Log aggregation cannot be started for container_e07_1503326514161_502342_01_000001, as its an absent container
> {noformat}
> Containers should not be allowed to disappear if they're not done being fully processed by the NM.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org