You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Fritz Budiyanto <fb...@icloud.com> on 2021/06/24 15:07:33 UTC
Task manager got stuck not restarting due to OOM: Metaspace
Hi,
I have a job which kept on restarting due to a bug, and it brought down the task manager with it due to OOM Metaspace. Please ignore the memory leak for a moment, the problem here is the task manager does not restart and hung which reduce the overall slots capacity. We are running Flink in Kubernetes and task manager stuck is a problem as it reduces cluster capacity.
Any idea how to make task manager restart upon OOM ? Here is the logs of task manager:
2021-06-24 08:59:30,437 INFO org.apache.kafka.clients.producer.KafkaProducer [] - [Producer clientId=producer-2] Closing the Kafka producer with timeoutMillis = 0 ms.
2021-06-24 08:59:30,437 INFO org.apache.kafka.clients.producer.KafkaProducer [] - [Producer clientId=producer-2] Proceeding to force close the producer since pending requests could not be completed within timeout 0 ms.
2021-06-24 08:59:32,200 ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler [] - FATAL: Thread 'AsyncOperations-thread-8' produced an uncaught exception. Stopping the process...
java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
2021-06-24 08:59:41,759 INFO org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager [] - Shutting down TaskExecutorLocalStateStoresManager.
2021-06-24 08:59:46,560 INFO org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting down BLOB cache
2021-06-24 08:59:45,414 INFO org.apache.flink.runtime.blob.PermanentBlobCache [] - Shutting down BLOB cache
2021-06-24 08:59:59,383 INFO org.apache.flink.runtime.filecache.FileCache [] - removed file cache directory /tmp/flink-dist-cache-bb7818d6-a691-4121-9fdb-670b7d4182c5
—
Fritz
Re: Task manager got stuck not restarting due to OOM: Metaspace
Posted by Fritz Budiyanto <fb...@icloud.com>.
I found this configuration which control jvm exit. Let me try it out:
taskmanager.jvm-exit-on-oom: true
> On Jun 24, 2021, at 8:07 AM, Fritz Budiyanto <fb...@icloud.com> wrote:
>
> Hi,
>
>
> I have a job which kept on restarting due to a bug, and it brought down the task manager with it due to OOM Metaspace. Please ignore the memory leak for a moment, the problem here is the task manager does not restart and hung which reduce the overall slots capacity. We are running Flink in Kubernetes and task manager stuck is a problem as it reduces cluster capacity.
> Any idea how to make task manager restart upon OOM ? Here is the logs of task manager:
>
>
> 2021-06-24 08:59:30,437 INFO org.apache.kafka.clients.producer.KafkaProducer [] - [Producer clientId=producer-2] Closing the Kafka producer with timeoutMillis = 0 ms.
> 2021-06-24 08:59:30,437 INFO org.apache.kafka.clients.producer.KafkaProducer [] - [Producer clientId=producer-2] Proceeding to force close the producer since pending requests could not be completed within timeout 0 ms.
> 2021-06-24 08:59:32,200 ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler [] - FATAL: Thread 'AsyncOperations-thread-8' produced an uncaught exception. Stopping the process...
> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
> 2021-06-24 08:59:41,759 INFO org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager [] - Shutting down TaskExecutorLocalStateStoresManager.
> 2021-06-24 08:59:46,560 INFO org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting down BLOB cache
> 2021-06-24 08:59:45,414 INFO org.apache.flink.runtime.blob.PermanentBlobCache [] - Shutting down BLOB cache
> 2021-06-24 08:59:59,383 INFO org.apache.flink.runtime.filecache.FileCache [] - removed file cache directory /tmp/flink-dist-cache-bb7818d6-a691-4121-9fdb-670b7d4182c5
>
> —
> Fritz
>