You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "David Mollitor (Jira)" <ji...@apache.org> on 2019/09/26 16:25:00 UTC
[jira] [Comment Edited] (MAPREDUCE-7243) Randomize List of
Resources to Localize
[ https://issues.apache.org/jira/browse/MAPREDUCE-7243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938756#comment-16938756 ]
David Mollitor edited comment on MAPREDUCE-7243 at 9/26/19 4:24 PM:
--------------------------------------------------------------------
Hmm, just randomizing the list here probably won't get the desired behavior. Need to investigate further. The list get created once and then passed around to each attempt.
https://github.com/apache/hadoop/blob/499c70eda5f6315cf7e13a4b3bbefc7e187bc457/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java#L1100-L1105
was (Author: belugabehr):
Hmm, just randomizing the list here probably won't get the desired behavior. Need to investigate further.
> Randomize List of Resources to Localize
> ---------------------------------------
>
> Key: MAPREDUCE-7243
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7243
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: nodemanager, performance
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Minor
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/LocalResourceBuilder.java
> Add a new parameter to {{LocalResourceBuilder}} that allows the list of resources to be shuffled randomly. This will allow the Localizer to spread the load of requests so that not all of the NodeManagers are requesting to localize the same files, in the same order, from the same DataNodes,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org