[jira] [Commented] (FLINK-4427) Implement container releasing logic (Standalone / Yarn / Mesos)

["Eron Wright (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093770#comment-16093770 ] 

Eron Wright  commented on FLINK-4427:
-------------------------------------

I think we should decide on one of two high-level strategies for scaling (up- or down-) the slots.   One is to rely on the JobManager, who has the most knowledge about the requirements and current performance of the job.  The other is to implement pooling strategies in the RM.   I prefer the former approach because it would tend to optimize Flink for one-job-per-cluster with good elasticity and locality while avoiding the duplication of scheduler logic that is best left to YARN/Mesos.

My suggestion is:
1. move the slotpool to a resource manager subclass intended for session mode only.   Treat the slotpool as the 'free pool'.
2. eagerly acquire a configured number of slots and keep them for the length of the session.
3. rework the RM-JM protocol to give the JM more information about capacity (based on YARN/Mesos specifics, e.g. offers and inverse-offers) and emphasize JM control over scaling (as opposed to implementing scaling logic at the RM level). 


> Implement container releasing logic (Standalone / Yarn / Mesos)
> ---------------------------------------------------------------
>
>                 Key: FLINK-4427
>                 URL: https://issues.apache.org/jira/browse/FLINK-4427
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Cluster Management
>            Reporter: Kurt Young
>
> Currently we only have allocation logic for SlotManager / ResourceManager, for some batch job, slots that already finished can be released, thus should trigger container release in different cluster modes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)