You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Mingliang Liu (Jira)" <ji...@apache.org> on 2020/09/22 19:01:00 UTC
[jira] [Resolved] (MAPREDUCE-7294) Only application master should
upload resource to Yarn Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mingliang Liu resolved MAPREDUCE-7294.
--------------------------------------
Fix Version/s: 2.10.2
Resolution: Fixed
Committed to all fix versions. Thanks for your contribution, [~wzzdreamer]. Thanks for your discussion and review [~jiwq]
[~iwasakims] Can we have this in Hadoop 2.10.1 release? Is that frozen for commits? Thanks!
> Only application master should upload resource to Yarn Shared Cache
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-7294
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7294
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.4
> Reporter: zhenzhao wang
> Assignee: zhenzhao wang
> Priority: Major
> Fix For: 3.2.2, 3.4.0, 3.1.5, 3.3.1, 2.10.2
>
>
> The design of yarn shared cache manager is only to allow application master should upload the jar/files/resource. However, there was a bug in the code since 2.9.0. Every node manager that take the job task will try to upload the jar/resources. Let's say one job have 5000 tasks. Then there will be up to 5000 NMs try to upload the jar. This is like DDOS and create a snowball effect. It will end up with inavailability of yarn shared cache manager. It wil cause time out in localization and lead to job failure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org