[jira] [Commented] (FLINK-20332) Add workers recovered from previous attempt to pending resources

["Xintong Song (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17287613#comment-17287613 ] 

Xintong Song commented on FLINK-20332:
--------------------------------------

To add recovered workers to pending resources, we need to get the {{TaskExecutorProcessSpec}} or {{WorkerResourceSpec}} of the recovered workers before they register to RM. There are several options to do this.
# Parse from the TM starting command, which contains TM resource specifications as dynamic properties. Starting command of recovered workers are only available for the native k8s deployment.
# Attach (serialized) resource specifications to workers as metadata: k8s annotations, yarn allocation tags (version 3.1+, at the price handling different versions with reflection and yarn RM keeps the tags in memory).
# Maintaining worker resource specs in HA, at the price of more HA space (linear to # of living TMs) and IO cost during TM allocation/release.

2) does not provide much extra benefit compared to 1), and 3) might be a bit too heavy. Therefore, as a first step, I propose to do this improvement with 1), for native K8s deployment only.
- Yarn can be supported with 2) later, if needed. Recovering resource specifications with different approach for Kubernetes / Yarn shouldn't be a problem.
- ATM, we do not support TMs with different resources on Mesos. Thus, TM resource specifications on Mesos should always be the default.

> Add workers recovered from previous attempt to pending resources
> ----------------------------------------------------------------
>
>                 Key: FLINK-20332
>                 URL: https://issues.apache.org/jira/browse/FLINK-20332
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Coordination
>            Reporter: Xintong Song
>            Assignee: Xintong Song
>            Priority: Major
>
> For active deployments (Native K8s/Yarn/Mesos), after a JM failover, workers from previous attempt should register to the new JM. Depending on the order that slot requests and TM registrations arrive at the RM, it could happen that RM allocates unnecessary new resources while there are recovered resources that can be reused.
> A potential improvement is to add recovered workers to pending resources, so that RM knows what resources are expected to be available soon and decide whether to allocate new resources accordingly.
> See also the discussion in FLINK-20249.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)