[jira] [Commented] (YARN-7018) Interface for adding extra behavior to node heartbeats

["Manikandan R (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16571962#comment-16571962 ] 

Manikandan R commented on YARN-7018:
------------------------------------

[~jlowe] and I had a very preliminary offline discussion on approaches. Thank you [~jlowe] for explaining the requirements in detail.

Summary of the discussion: (Copying [~jlowe]'s suggestion as is)

The point of YARN-7018 is to create a hook that runs before the scheduler and, as much as possible, is *not* scheduler-dependent. If we have to do something scheduler-dependent then we might as well just modify the schedulers directly.

I haven't had much time to think about it, but one potential way to implement this is a very simple plugin API. Something like this:
 - Admin configures the plugin via a conf key
 - Plugin only has one API call, onNodeHeartbeat or something like that. It's called when a NODE_UPDATE event is processed and before the main scheduler loop is called
 - Call arguments would include the SchedulerNode and maybe the RMContext.

From the RMContext the plugin can get the scheduler and do scheduler-specific things if it wants to, or can be scheduler-specific if it wants to.

I'd rather not force plugin writers to implement every scheduler supported. It's a burden to writing a plugin and an obstacle to providing new schedulers in YARN.

> Interface for adding extra behavior to node heartbeats
> ------------------------------------------------------
>
>                 Key: YARN-7018
>                 URL: https://issues.apache.org/jira/browse/YARN-7018
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: resourcemanager
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Major
>
> This JIRA tracks an interface for plugging in new behavior to node heartbeat processing.  Adding a formal interface for additional node heartbeat processing would allow admins to configure new functionality that is scheduler-independent without needing to replace the entire scheduler.  For example, both YARN-5202 and YARN-5215 had approaches where node heartbeat processing was extended to implement new functionality that was essentially scheduler-independent and could be implemented as a plugin with this interface.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org