You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Naganarasimha G R (JIRA)" <ji...@apache.org> on 2016/08/17 14:10:22 UTC
[jira] [Commented] (YARN-574) PrivateLocalizer does not support
parallel resource download via ContainerLocalizer
[ https://issues.apache.org/jira/browse/YARN-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424574#comment-15424574 ]
Naganarasimha G R commented on YARN-574:
----------------------------------------
[~ojoshi], Seems like overall patch is fine except for the logic that in *ContainerLocalizer.localizeFiles* we can optimize to receive all the resources to be localized in one shot (already we receive it as list<ResourceLocalizationSpec> so just need to confirm the same in Resourceloclization service) and avoid polling/hb the server frequently just based on the number of threads in the Container Localizer.
Retriggering the build to see whether old patch works!
> PrivateLocalizer does not support parallel resource download via ContainerLocalizer
> -----------------------------------------------------------------------------------
>
> Key: YARN-574
> URL: https://issues.apache.org/jira/browse/YARN-574
> Project: Hadoop YARN
> Issue Type: Sub-task
> Affects Versions: 2.6.0, 2.8.0, 2.7.1
> Reporter: Omkar Vinit Joshi
> Assignee: Omkar Vinit Joshi
> Attachments: YARN-574.1.patch, YARN-574.2.patch
>
>
> At present private resources will be downloaded in parallel only if multiple containers request the same resource. However otherwise it will be serial. The protocol between PrivateLocalizer and ContainerLocalizer supports multiple downloads however it is not used and only one resource is sent for downloading at a time.
> I think we can increase / assure parallelism (even for single container requesting resource) for private/application resources by making multiple downloads per ContainerLocalizer.
> Total Parallelism before
> = number of threads allotted for PublicLocalizer [public resource] + number of containers[private and application resource]
> Total Parallelism after
> = number of threads allotted for PublicLocalizer [public resource] + number of containers * max downloads per container [private and application resource]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org