You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Aljoscha Krettek (Jira)" <ji...@apache.org> on 2020/05/05 08:12:00 UTC
[jira] [Updated] (FLINK-17514) TaskCancelerWatchdog does not kill
TaskManager
[ https://issues.apache.org/jira/browse/FLINK-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aljoscha Krettek updated FLINK-17514:
-------------------------------------
Description:
The watchdog reports a fatal error using {{taskManager.notifyFatalError(msg, null)}}. This should normally lead to the TaskManager being terminated. The code introduced in FLINK-16225
tries to look at the passed exception and will eventually fail with a {{NullPointerException}}, which prevents the TaskManager from being terminated.
Stacktrace:
{code:java}
2020-05-05 09:43:01,588 ERROR org.apache.flink.runtime.taskmanager.Task - Task did not exit gracefully within 180 + seconds.
2020-05-05 09:43:01,588 ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor - Task did not exit gracefully within 180 + seconds.
2020-05-05 09:43:01,588 ERROR org.apache.flink.runtime.taskmanager.Task - Error in Task Cancellation Watch Dog
java.lang.NullPointerException
at org.apache.flink.util.ExceptionUtils.isOutOfMemoryErrorWithMessageStartingWith(ExceptionUtils.java:186)
at org.apache.flink.util.ExceptionUtils.isMetaspaceOutOfMemoryError(ExceptionUtils.java:170)
at org.apache.flink.util.ExceptionUtils.enrichTaskManagerOutOfMemoryError(ExceptionUtils.java:144)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.onFatalError(TaskManagerRunner.java:249)
at org.apache.flink.runtime.taskexecutor.TaskExecutor$TaskManagerActionsImpl.notifyFatalError(TaskExecutor.java:1751)
at org.apache.flink.runtime.taskmanager.Task$TaskCancelerWatchDog.run(Task.java:1514)
at java.lang.Thread.run(Thread.java:748)
{code}
was:
The watchdog reports a fatal error using {{taskManager.notifyFatalError(msg, null)}}. This should normally lead to the TaskManager being terminated. The code introduced in FLINK-16255 tries to look at the passed exception and will eventually fail with a {{NullPointerException}}, which prevents the TaskManager from being terminated.
Stacktrace:
{code}
2020-05-05 09:43:01,588 ERROR org.apache.flink.runtime.taskmanager.Task - Task did not exit gracefully within 180 + seconds.
2020-05-05 09:43:01,588 ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor - Task did not exit gracefully within 180 + seconds.
2020-05-05 09:43:01,588 ERROR org.apache.flink.runtime.taskmanager.Task - Error in Task Cancellation Watch Dog
java.lang.NullPointerException
at org.apache.flink.util.ExceptionUtils.isOutOfMemoryErrorWithMessageStartingWith(ExceptionUtils.java:186)
at org.apache.flink.util.ExceptionUtils.isMetaspaceOutOfMemoryError(ExceptionUtils.java:170)
at org.apache.flink.util.ExceptionUtils.enrichTaskManagerOutOfMemoryError(ExceptionUtils.java:144)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.onFatalError(TaskManagerRunner.java:249)
at org.apache.flink.runtime.taskexecutor.TaskExecutor$TaskManagerActionsImpl.notifyFatalError(TaskExecutor.java:1751)
at org.apache.flink.runtime.taskmanager.Task$TaskCancelerWatchDog.run(Task.java:1514)
at java.lang.Thread.run(Thread.java:748)
{code}
> TaskCancelerWatchdog does not kill TaskManager
> ----------------------------------------------
>
> Key: FLINK-17514
> URL: https://issues.apache.org/jira/browse/FLINK-17514
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
> Affects Versions: 1.10.1, 1.11.0
> Reporter: Aljoscha Krettek
> Priority: Blocker
> Fix For: 1.10.1, 1.11.0
>
>
> The watchdog reports a fatal error using {{taskManager.notifyFatalError(msg, null)}}. This should normally lead to the TaskManager being terminated. The code introduced in FLINK-16225
> tries to look at the passed exception and will eventually fail with a {{NullPointerException}}, which prevents the TaskManager from being terminated.
> Stacktrace:
> {code:java}
> 2020-05-05 09:43:01,588 ERROR org.apache.flink.runtime.taskmanager.Task - Task did not exit gracefully within 180 + seconds.
> 2020-05-05 09:43:01,588 ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor - Task did not exit gracefully within 180 + seconds.
> 2020-05-05 09:43:01,588 ERROR org.apache.flink.runtime.taskmanager.Task - Error in Task Cancellation Watch Dog
> java.lang.NullPointerException
> at org.apache.flink.util.ExceptionUtils.isOutOfMemoryErrorWithMessageStartingWith(ExceptionUtils.java:186)
> at org.apache.flink.util.ExceptionUtils.isMetaspaceOutOfMemoryError(ExceptionUtils.java:170)
> at org.apache.flink.util.ExceptionUtils.enrichTaskManagerOutOfMemoryError(ExceptionUtils.java:144)
> at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.onFatalError(TaskManagerRunner.java:249)
> at org.apache.flink.runtime.taskexecutor.TaskExecutor$TaskManagerActionsImpl.notifyFatalError(TaskExecutor.java:1751)
> at org.apache.flink.runtime.taskmanager.Task$TaskCancelerWatchDog.run(Task.java:1514)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)