You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "Karthik Palaniappan (JIRA)" <ji...@apache.org> on 2018/04/04 23:12:00 UTC
[jira] [Created] (YARN-8118) Better utilize gracefully
decommissioning node managers
Karthik Palaniappan created YARN-8118:
-----------------------------------------
Summary: Better utilize gracefully decommissioning node managers
Key: YARN-8118
URL: https://issues.apache.org/jira/browse/YARN-8118
Project: Hadoop YARN
Issue Type: Sub-task
Components: yarn
Affects Versions: 2.8.2
Environment: * Google Compute Engine (Dataproc)
* Java 8
* Hadoop 2.8.2 using client-mode graceful decommissioning
Reporter: Karthik Palaniappan
Proposal design doc with background + details: (will link – please comment on that doc)
tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications to complete before shutting down, but they cannot run new containers from those in-progress applications. This is wasteful, particularly in environments where you are billed by resource usage (e.g. EC2).
Proposal: YARN should schedule containers from in-progress applications on DECOMMISSIONING nodes, but should still avoid scheduling containers from new applications. That will make in-progress applications complete faster and let nodes decommission faster. Overall, this should be cheaper.
I have a working patch without unit tests (will attach) that's surprisingly just a few real lines of code. If folks are happy with the proposal, I'll write unit tests and also write a patch targeted at trunk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org