You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Till Rohrmann (Jira)" <ji...@apache.org> on 2019/10/24 14:25:00 UTC

[jira] [Commented] (FLINK-14074) MesosResourceManager can't create new taskmanagers in Session Cluster Mode.

    [ https://issues.apache.org/jira/browse/FLINK-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16958931#comment-16958931 ] 

Till Rohrmann commented on FLINK-14074:
---------------------------------------

I think the problem is the following: Flink thinks that it starts a {{TaskExecutor}} with 1 GB of memory. However, for some reason, the {{TaskExecutor}} is started with more memory. Due to this, the reported slots will have more resources than Flink expects. Internally, Flink registers pending slots in the {{SlotManager#pendingSlots}} [1]. The contract is that a pending slot gets removed if Flink receives an slot report which exactly matches the pending slot's profile [2]. If this is not the case (e.g. if the {{TaskExecutor}} is started with more memory than asked for), then Flink still accepts the reported slot but it does not remove the pending slot. As a consequence, Flink still thinks that there will be a slot coming which will exactly match the pending slot. Hence, when you submit the second job and if the first {{TaskExecutor}} has been released, then Flink thinks that it does not need to start a new {{TaskExecutor}} because there is still a pending slot in {{SlotManagerImpl#pendingSlots}}.

A similar problem has been reported here [3].

A quick solution would be to figure out why the {{MesosResourceManager}} starts {{TaskExecutors}} with more memory. A cleaner and more robust solution would probably be to not rely on exact {{ResourceProfile}} matchings in order to remove pending slots. It would be better if one could set the {{TaskExecutor's}} {{ResourceID}} and match against this instead of the {{ResourceProfile}}.

[1] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java#L801
[2] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java#L594
[3] https://stackoverflow.com/questions/58537199/apache-flink-resource-manager-app-master-fails-allocating-new-task-managers-af

> MesosResourceManager can't create new taskmanagers in Session Cluster Mode.
> ---------------------------------------------------------------------------
>
>                 Key: FLINK-14074
>                 URL: https://issues.apache.org/jira/browse/FLINK-14074
>             Project: Flink
>          Issue Type: Bug
>          Components: Deployment / Mesos
>    Affects Versions: 1.9.0
>         Environment: Flink HA Session cluster 1.9.0 on mesos.
>            Reporter: Alexander Kasyanenko
>            Priority: Critical
>
> Hi, I'm trying to launch multiple jobs in Flink Session Cluster, deployed on mesos.
>  Flink's version is 1.9.0.
> The very first resource allocation completes successfully, and first submitted job launches, but submitting any amount of jobs afterwords doesn't affect the cluster in any way and no additional TaskManagers are allocated.
> From the logs I see that MesosResourceManager is requesting Slots for the newly submitted jobs:  "{{o.a.f.m.r.c.MesosResourceManager - Request slot with profile ResourceProfile..."}} but line {{"Starting a new worker.}}" appears in log only the same amount of times as taskmanagers count, allocated for the first job.
> I'm a complete noob in flink internals, but took a wild guess about a reason. I think that the problem is in this check: [https://github.com/apache/flink/blob/release-1.9.0/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java#L436]
> It might be that RM is lazily allocated at the first call by a factory, and then a private final field {{slotsPerWorker}} is set. So this check will prevent creation of any new worker after iterator traverses the entire collection. My main assumption is that {{slotsPerWorker}} is never modified again.
>  
> I'm sorry that I didn't do much of investigation before reporting, but I'll try to do some after a weekend. I plan to build flink without this check and see if it helps. Also I'll play around with tests for this RM. Since it's my time running time flink internals, I'll be back after a few days.
> Any help will much appreciated.
> Thanks in advance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)