You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Abhinaba Sarkar (Jira)" <ji...@apache.org> on 2021/03/02 04:58:00 UTC
[jira] [Updated] (YARN-10661) Add option to scale down specific
containers in flex API
[ https://issues.apache.org/jira/browse/YARN-10661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhinaba Sarkar updated YARN-10661:
-----------------------------------
Description:
We can flex the number of containers in yarn-service. But currently we do not have option to specify which containers are to be scaled down. This is useful in scaling down Hive LLAP clusters. For example -
{code:java}
2021-02-22 12:52:58,983 [IPC Server handler 0 on 24799] INFO service.ClientAMService - Flexing component llap to 42021-02-22 12:52:58,983 [Component dispatcher] INFO component.Component - [FLEX DOWN COMPONENT llap]: scaling down from 6 to 42021-02-22 12:52:58,984 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE llap-5 : container_e02_1613896771789_0071_01_000007]: Flexed down by user, destroying.2021-02-22 12:52:58,987 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE llap-4 : container_e02_1613896771789_0071_01_000006]: Flexed down by user, destroying.
{code}
Last added containers are first ones to be destroyed. This might not be the most optimal approach as we would like the least loaded(in terms of executors) containers to be destroyed first.
To enable this, we propose an option to add the list of containers that can be scaled down by flexing.
was:
We can flex the number of containers in yarn-service. But currently we do not have option to specify which containers are to be scaled down. This is useful in scaling down Hive LLAP clusters. For example -
{code:java}
2021-02-22 12:52:58,983 [IPC Server handler 0 on 24799] INFO service.ClientAMService - Flexing component llap to 42021-02-22 12:52:58,983 [Component dispatcher] INFO component.Component - [FLEX DOWN COMPONENT llap]: scaling down from 6 to 42021-02-22 12:52:58,984 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE llap-5 : container_e02_1613896771789_0071_01_000007]: Flexed down by user, destroying.2021-02-22 12:52:58,987 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE llap-4 : container_e02_1613896771789_0071_01_000006]: Flexed down by user, destroying.
{code}
Last added containers are first ones to be destroyed. This might not be the most optimal approach as we would like the least loaded(in terms of executors) containers to be destroyed first. We propose an option to add the list of containers that can be scaled down by flexing.
> Add option to scale down specific containers in flex API
> --------------------------------------------------------
>
> Key: YARN-10661
> URL: https://issues.apache.org/jira/browse/YARN-10661
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Reporter: Abhinaba Sarkar
> Priority: Major
>
> We can flex the number of containers in yarn-service. But currently we do not have option to specify which containers are to be scaled down. This is useful in scaling down Hive LLAP clusters. For example -
> {code:java}
> 2021-02-22 12:52:58,983 [IPC Server handler 0 on 24799] INFO service.ClientAMService - Flexing component llap to 42021-02-22 12:52:58,983 [Component dispatcher] INFO component.Component - [FLEX DOWN COMPONENT llap]: scaling down from 6 to 42021-02-22 12:52:58,984 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE llap-5 : container_e02_1613896771789_0071_01_000007]: Flexed down by user, destroying.2021-02-22 12:52:58,987 [Component dispatcher] INFO instance.ComponentInstance - [COMPINSTANCE llap-4 : container_e02_1613896771789_0071_01_000006]: Flexed down by user, destroying.
> {code}
> Last added containers are first ones to be destroyed. This might not be the most optimal approach as we would like the least loaded(in terms of executors) containers to be destroyed first.
> To enable this, we propose an option to add the list of containers that can be scaled down by flexing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org