You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "David Mollitor (Jira)" <ji...@apache.org> on 2019/09/26 16:24:00 UTC
[jira] [Issue Comment Deleted] (MAPREDUCE-7243) Randomize List of
Resources to Localize
[ https://issues.apache.org/jira/browse/MAPREDUCE-7243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Mollitor updated MAPREDUCE-7243:
--------------------------------------
Comment: was deleted
(was: I was thinking that the list gets generated once and then provided to each task. So the list is random, but all the tasks load the files still in the same order. However, it looks like the list is generated for each Task Attempt.
https://github.com/apache/hadoop/blob/499c70eda5f6315cf7e13a4b3bbefc7e187bc457/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java#L913)
> Randomize List of Resources to Localize
> ---------------------------------------
>
> Key: MAPREDUCE-7243
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7243
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: nodemanager, performance
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Minor
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/LocalResourceBuilder.java
> Add a new parameter to {{LocalResourceBuilder}} that allows the list of resources to be shuffled randomly. This will allow the Localizer to spread the load of requests so that not all of the NodeManagers are requesting to localize the same files, in the same order, from the same DataNodes,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org