You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Piotr Nowojski (Jira)" <ji...@apache.org> on 2021/03/18 16:11:00 UTC
[jira] [Comment Edited] (FLINK-21821) Clean up Task/StreamTask
cancellation and remove unnecessary concurrency
[ https://issues.apache.org/jira/browse/FLINK-21821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304246#comment-17304246 ]
Piotr Nowojski edited comment on FLINK-21821 at 3/18/21, 4:10 PM:
------------------------------------------------------------------
Thanks for rising this concern [~kezhuw]. However I don't fully agree that cancel should only do "flag things but not closing stuff". Our cancellation has a couple of steps:
1. Quick cancellation that leaves JVM in tact ({{TaskCanceler}})
a) cancel main loop (mailbox or legacy source task), and set {{running = false}} to ignore all exceptions from here on
b) close our resources where a thread might block on. For example {{LocalBufferPool}}
c) Interrupt the thread, in case it is in a blocking call.
2. {{TaskInterrupter}} thread initiates interrupts after a timeout, in a last ditch effort before killing the JVM
3. {{TaskCancelerWatchDog}} kills JVM if {{Task}} doesn't stop
1 b) can help us answer your concerns. For example for historical reasons, I think we are closing {{InputGate}}s, because they were blocking in the past. Now they are non-blocking (at least for the {{StreamTask}}), so we don't need to pollute {{SingleInputGate}} with for example {{CompletableFuture closeFuture}}. The same *might* apply to other similar things on the input side like {{AtomicBoolean RemoteInputChannel#isReleased}}, but it would have to be carefully considered. Definitely removing those, would simplify threading model of those classes.
On the other hand, closing of {{LocalBufferPool}} would need to kept as it is, as long as we can block inside it. I don't know about {{MemoryManager}} or other places.
There is one more interesting bit. It's important to not issue interrupts when {{Task/StreamTask}} is already closing/releasing resources. Otherwise some interrupt issued during releasing resources could lead to a resource leak. We are already guarding against that via setting {{AbstractInvokable#setShouldInterruptOnCancel}} at the right place.
was (Author: pnowojski):
Thanks for rising this concern [~kezhuw]. However I don't fully agree that cancel should only do "flag things but not closing stuff". Our cancellation has a couple of steps:
1. Quick cancellation that leaves JVM in tact ({{TaskCanceler}})
a) cancel main loop (mailbox or legacy source task), and set {{running = false}} to ignore all exceptions from here on
b) close our resources where a thread might block on. For example {{LocalBufferPool}}
c) Interrupt the thread, in case it is in a blocking call.
2. {{TaskInterrupter}} thread initiates interrupts after a timeout, in a last ditch effort before killing the JVM
3. {{TaskCancelerWatchDog}} kills JVM if {{Task}} doesn't stop
1 b) can help us answer your concerns. For example for historical reasons, I think we are closing {{InputGate}}s, because they were blocking in the past. Now they are non-blocking (at least for the {{StreamTask}}), so we don't need to pollute {{SingleInputGate}} with for example {{CompletableFuture closeFuture}}. The same *might* apply to other similar things on the input side like {{AtomicBoolean RemoteInputChannel#isReleased}}, but it would have to be carefully considered. Definitely removing those, would simplify threading model of those classes.
On the other hand, closing of {{LocalBufferPool}} would need to kept as it is, as long as we can block inside it. I don't know about {{MemoryManager}} or other places.
> Clean up Task/StreamTask cancellation and remove unnecessary concurrency
> -------------------------------------------------------------------------
>
> Key: FLINK-21821
> URL: https://issues.apache.org/jira/browse/FLINK-21821
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network, Runtime / Task
> Affects Versions: 1.13.0
> Reporter: Kezhu Wang
> Priority: Major
>
> I think {{AbstractInvokable.cancel}} should only do flag things but not closing staff. Otherwise task canceler and task thread will contend on these closing tasks.
> It is same for {{BatchTask.cancel}} which might be confirmed as one potential cause for FLINK-21728.
> cc [~dwysakowicz] [~roman_khachatryan] [~pnowojski] [~AHeise]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)