You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Kuhu Shukla (JIRA)" <ji...@apache.org> on 2017/09/27 12:07:00 UTC
[jira] [Comment Edited] (YARN-7244) ShuffleHandler is not aware of
disks that are added
[ https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181059#comment-16181059 ]
Kuhu Shukla edited comment on YARN-7244 at 9/27/17 12:06 PM:
-------------------------------------------------------------
Thank you [~bibinchundatt] for the review comments!
bq. Better to check directory exists first if we are not concerned of permission . thoughts?
Makes sense to me.
bq. Testcase is successful even if YARN_SHUFFLE_BAD_DIRS_FILTER_ENABLED set to false.
That is expected. This config essentially decides if we should stick to existing behavior (value=true) of removing directories if they are bad or not (value=false).
was (Author: kshukla):
bq. Better to check directory exists first if we are not concerned of permission . thoughts?
Makes sense to me.
bq. Testcase is successful even if YARN_SHUFFLE_BAD_DIRS_FILTER_ENABLED set to false.
That is expected. This config essentially decides if we should stick to existing behavior (value=true) of removing directories if they are bad or not (value=false).
> ShuffleHandler is not aware of disks that are added
> ---------------------------------------------------
>
> Key: YARN-7244
> URL: https://issues.apache.org/jira/browse/YARN-7244
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Attachments: YARN-7244.001.patch, YARN-7244.002.patch
>
>
> The ShuffleHandler permanently remembers the list of "good" disks on NM startup. If disks later are added to the node then map tasks will start using them but the ShuffleHandler will not be aware of them. The end result is that the data cannot be shuffled from the node leading to fetch failures and re-runs of the map tasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org