You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "TisonKun (JIRA)" <ji...@apache.org> on 2018/12/05 13:32:00 UTC
[jira] [Updated] (FLINK-11078) Capability to define the numerical
range for running TaskExecutors
[ https://issues.apache.org/jira/browse/FLINK-11078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
TisonKun updated FLINK-11078:
-----------------------------
Description:
In pre-FLIP-6 context, we start a yarn session with fixed number of running {{TaskManagers}}. This is good for user to run a series of small jobs on a specific cluster and reduce the cost of deploying a cluster per job. In current code base, we start a yarn session with none pre-allocated TMs, but allocates them on a certain job submitted and running.
To get benefits from both mode, I propose introducing a pair {{(min, max)}} represents the minimum and maximum for the number of running {{TaskExecutors}}.
With such option, when setting minimum = maximum = n we effectively have the same behaviour as before with the pre-Flip-6 code; and when setting minimum = 0, maximum = inf we effectively have the same behaviour as current code path.
Most of the implementation area would be passing such option in {{FlinkYarnSessionCli}} and respecting it in {{ResourceManager}} on {{grantLeadership}} and {{stopWorker}}.
Hopefully the changes are not that big and I would try to draft more details, and be glad to your suggestions and ideas.
> Capability to define the numerical range for running TaskExecutors
> ------------------------------------------------------------------
>
> Key: FLINK-11078
> URL: https://issues.apache.org/jira/browse/FLINK-11078
> Project: Flink
> Issue Type: New Feature
> Components: Client, ResourceManager
> Affects Versions: 1.8.0
> Reporter: TisonKun
> Assignee: TisonKun
> Priority: Major
> Fix For: 1.8.0
>
>
> In pre-FLIP-6 context, we start a yarn session with fixed number of running {{TaskManagers}}. This is good for user to run a series of small jobs on a specific cluster and reduce the cost of deploying a cluster per job. In current code base, we start a yarn session with none pre-allocated TMs, but allocates them on a certain job submitted and running.
> To get benefits from both mode, I propose introducing a pair {{(min, max)}} represents the minimum and maximum for the number of running {{TaskExecutors}}.
> With such option, when setting minimum = maximum = n we effectively have the same behaviour as before with the pre-Flip-6 code; and when setting minimum = 0, maximum = inf we effectively have the same behaviour as current code path.
> Most of the implementation area would be passing such option in {{FlinkYarnSessionCli}} and respecting it in {{ResourceManager}} on {{grantLeadership}} and {{stopWorker}}.
> Hopefully the changes are not that big and I would try to draft more details, and be glad to your suggestions and ideas.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)