[jira] [Comment Edited] (FLINK-20605) DeclarativeSlotManager crashes if slot allocation notification is processed after taskexecutor shutdown

["Chesnay Schepler (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249605#comment-17249605 ] 

Chesnay Schepler edited comment on FLINK-20605 at 12/15/20, 10:06 AM:
----------------------------------------------------------------------

I think what happens is that, when the TaskExecutor sends the allocation update and un-register messages quickly in succession, it can happen that the the requestFuture in {{DeclarativeSlotManager#allocateSlot}} is completed while the un-register message is already queued for being processed. This causes the handleAsync to be run after the un-register message.
That's my guess anyway.

I have seen 2 more failure cases in this method so far; 1 where we process stuff after already having shut down, 1 where we are informed about a slot being allocated, but it already was allocated.



was (Author: zentol):
I think what happens is that when the TaskExecutor sends the allocation update and un-register messages quickly in succession it can happen that the the requestFuture in {{DeclarativeSlotManager#allocateSlot}} is completed while the un-register message is already queued for being processed. This causes the handleAsync to be run after the un-register message.
That's my guess anyway.

I have seen 2 more failure cases in this method so far; 1 where we process stuff after already having shut down, 1 where we are informed about a slot being allocated, but it already was allocated.


> DeclarativeSlotManager crashes if slot allocation notification is processed after taskexecutor shutdown
> -------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-20605
>                 URL: https://issues.apache.org/jira/browse/FLINK-20605
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Coordination
>    Affects Versions: 1.13.0
>            Reporter: Chesnay Schepler
>            Assignee: Chesnay Schepler
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.13.0
>
>
> It is possible that a notification from a task executor about a slot being allocated can be processed after that very task executor has unregistered itself from the resource manager.
> As a result we run into an exception when trying to mark this slot as allocated, because it no longer exists and a precondition catches this case.
> We could solve this by checking in {{DeclarativeResourceManager#allocateSlot}} whether the task executor we received the acknowledge from is still registered.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)