[jira] [Comment Edited] (FLINK-14074) MesosResourceManager can't create new taskmanagers in Session Cluster Mode.

["Alexander Kasyanenko (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932426#comment-16932426 ] 

Alexander Kasyanenko edited comment on FLINK-14074 at 9/18/19 3:41 PM:
-----------------------------------------------------------------------

Hi [~till.rohrmann], I'm sorry for the delay on my end. I was able to look into this issue only today.

Firstly, I've prepared example logs, but I need this log to be validated (still in progress), so I can share it outside of company.

But anyway, I've built flink with extended logging and found out some interesting things.
 1. This check: [https://github.com/apache/flink/blob/release-1.9.0/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java#L436] has nothing to do with the problem, it isn't even called after the first job is cancelled/completed.

2. Last thing which happens during new slot allocation is this slot request submitted here: [https://github.com/apache/flink/blob/release-1.9.0/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java#L437], which I traced to method {{internalRequestSlot}} [https://github.com/apache/flink/blob/release-1.9.0/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java#L750] but method {{allocateResource}} was never called: [https://github.com/apache/flink/blob/release-1.9.0/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java#L795]

 

Right now I'm adding even more logging to see what is blocking slot allocation. I'll keep you posted, but as an update, I'm leaning towards some problems with pending slots.

 

UPD. I've rerun my test-case one more time and been able to locate possible problem even further. For some reason when first job is completed or cancelled, it still leaves pending slots behind, even if UI shows 0 total task slots. So when the next job is submitted, it is successfully getting a slot here: [https://github.com/apache/flink/blob/release-1.9.0/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java#L779] (because {{pendingSlots}} map is not empty) and acquiring it here: [https://github.com/apache/flink/blob/release-1.9.0/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java#L814] . But since mesos worker are already stopped this slot acquiring freezes job execution and every task ends up in SCHEDULED state. 
Since I haven't been able to find a place where {{pendingSlots}} map is cleaned after a worker shutdown,  I think that might be a reason for such behaviour.

[~till.rohrmann] , any ideas here?


was (Author: atlaster):
Hi [~till.rohrmann], I'm sorry for the delay on my end. I was able to look into this issue only today.

Firstly, I've prepared example logs, but I need this log to be validated (still in progress), so I can share it outside of company.

But anyway, I've built flink with extended logging and found out some interesting things.
1. This check: [https://github.com/apache/flink/blob/release-1.9.0/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java#L436] has nothing to do with the problem, it isn't even called after the first job is cancelled/completed.

2. Last thing which happens during new slot allocation is this slot request submitted here: [https://github.com/apache/flink/blob/release-1.9.0/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java#L437], which I traced to method {{internalRequestSlot}} [https://github.com/apache/flink/blob/release-1.9.0/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java#L750] but method {{allocateResource}} was never called: [https://github.com/apache/flink/blob/release-1.9.0/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java#L795]

 

Right now I'm adding even more logging to see what is blocking slot allocation. I'll keep you posted, but as an update, I'm leaning towards some problems with pending slots.

> MesosResourceManager can't create new taskmanagers in Session Cluster Mode.
> ---------------------------------------------------------------------------
>
>                 Key: FLINK-14074
>                 URL: https://issues.apache.org/jira/browse/FLINK-14074
>             Project: Flink
>          Issue Type: Bug
>          Components: Deployment / Mesos
>    Affects Versions: 1.9.0
>         Environment: Flink HA Session cluster 1.9.0 on mesos.
>            Reporter: Alexander Kasyanenko
>            Priority: Major
>
> Hi, I'm trying to launch multiple jobs in Flink Session Cluster, deployed on mesos.
>  Flink's version is 1.9.0.
> The very first resource allocation completes successfully, and first submitted job launches, but submitting any amount of jobs afterwords doesn't affect the cluster in any way and no additional TaskManagers are allocated.
> From the logs I see that MesosResourceManager is requesting Slots for the newly submitted jobs:  "{{o.a.f.m.r.c.MesosResourceManager - Request slot with profile ResourceProfile..."}} but line {{"Starting a new worker.}}" appears in log only the same amount of times as taskmanagers count, allocated for the first job.
> I'm a complete noob in flink internals, but took a wild guess about a reason. I think that the problem is in this check: [https://github.com/apache/flink/blob/release-1.9.0/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java#L436]
> It might be that RM is lazily allocated at the first call by a factory, and then a private final field {{slotsPerWorker}} is set. So this check will prevent creation of any new worker after iterator traverses the entire collection. My main assumption is that {{slotsPerWorker}} is never modified again.
>  
> I'm sorry that I didn't do much of investigation before reporting, but I'll try to do some after a weekend. I plan to build flink without this check and see if it helps. Also I'll play around with tests for this RM. Since it's my time running time flink internals, I'll be back after a few days.
> Any help will much appreciated.
> Thanks in advance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)