You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Zhankun Tang (JIRA)" <ji...@apache.org> on 2018/11/06 03:31:00 UTC
[jira] [Comment Edited] (YARN-8851) [Umbrella] A new pluggable
device plugin framework to ease vendor plugin development
[ https://issues.apache.org/jira/browse/YARN-8851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16676074#comment-16676074 ]
Zhankun Tang edited comment on YARN-8851 at 11/6/18 3:30 AM:
-------------------------------------------------------------
{quote}1) Regarding to the NM_PLUGGABLE_DEVICE_FRAMEWORK_PREFER_CUSTOMIZED_SCHEDULER, should we just use default scheduler if device plugin doesn't provide their customized scheduler? We should assume that load device plugin runs "trusted" code, we may not need to add extra protection here.
{quote}
Zhankun–> Agree.
{quote}2) DeviceSchedulerManager, it sounds like "manages scheduler", however it handles how to map device to containers, and scheduler is just implementation details. How about call it DeviceMappingManager?
{quote}
-
{quote}internalAssignDevices should be private, and it is a bit long, might be better for future maintenance if you can break it down to multiple methods.
{quote}
Zhankun -> Good idea. Will do that.
{quote}I think we could move to make this POC to sub tasks and get them done piece by piece. It gonna be helpful if you can highlight subtasks required.
{quote}
Zhankun-> The YARN-8880, YARN-8881, YARN-8882, YARN-8883, YARN-8885 are our Phase 1 highlighted subtasks. They'll be marked "InProgress" to highlight.
Thanks for the review! [~leftnoteasy]
was (Author: tangzhankun):
{quote}1) Regarding to the NM_PLUGGABLE_DEVICE_FRAMEWORK_PREFER_CUSTOMIZED_SCHEDULER, should we just use default scheduler if device plugin doesn't provide their customized scheduler? We should assume that load device plugin runs "trusted" code, we may not need to add extra protection here.
{quote}
Zhankun–> Agree.
{quote}2) DeviceSchedulerManager, it sounds like "manages scheduler", however it handles how to map device to containers, and scheduler is just implementation details. How about call it DeviceMappingManager?
{quote}
-
{quote}internalAssignDevices should be private, and it is a bit long, might be better for future maintenance if you can break it down to multiple methods.
{quote}
Zhankun -> Good idea. Will do that.
{quote}I think we could move to make this POC to sub tasks and get them done piece by piece. It gonna be helpful if you can highlight subtasks required.
{quote}
Zhankun-> The YARN-8880, YARN-8881, YARN-8882, YARN-8883, YARN-8885 are our Phase 1 highlighted subtasks.
Thanks for the review! [~leftnoteasy]
> [Umbrella] A new pluggable device plugin framework to ease vendor plugin development
> ------------------------------------------------------------------------------------
>
> Key: YARN-8851
> URL: https://issues.apache.org/jira/browse/YARN-8851
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: yarn
> Reporter: Zhankun Tang
> Assignee: Zhankun Tang
> Priority: Major
> Attachments: YARN-8851-WIP2-trunk.001.patch, YARN-8851-WIP3-trunk.001.patch, YARN-8851-WIP4-trunk.001.patch, YARN-8851-WIP5-trunk.001.patch, YARN-8851-WIP6-trunk.001.patch, YARN-8851-WIP7-trunk.001.patch, YARN-8851-WIP8-trunk.001.patch, YARN-8851-WIP9-trunk.001.patch, YARN-8851-trunk.001.patch, [YARN-8851] YARN_New_Device_Plugin_Framework_Design_Proposal-3.pdf, [YARN-8851] YARN_New_Device_Plugin_Framework_Design_Proposal-4.pdf, [YARN-8851] YARN_New_Device_Plugin_Framework_Design_Proposal.pdf
>
>
> At present, we support GPU/FPGA device in YARN through a native, coupling way. But it's difficult for a vendor to implement such a device plugin because the developer needs much knowledge of YARN internals. And this brings burden to the community to maintain both YARN core and vendor-specific code.
> Here we propose a new device plugin framework to ease vendor device plugin development and provide a more flexible way to integrate with YARN NM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org