You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Niels Basjes <Ni...@basjes.nl> on 2020/03/02 11:16:12 UTC

[Kubernetes] java.lang.OutOfMemoryError: Metaspace ?

Hi,

I'm running a lot of batch jobs on Kubernetes once in a while I get this
exception.
What is causing this?
How can I fix this?

Niels Basjes

java.lang.OutOfMemoryError: Metaspace
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:
142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.apache.flink.util.ChildFirstClassLoader.loadClass(
ChildFirstClassLoader.java:60)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:
142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.apache.flink.util.ChildFirstClassLoader.loadClass(
ChildFirstClassLoader.java:60)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:60)
    at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(
ESLoggerFactory.java:45)
    at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(
ESLoggerFactory.java:53)
    at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:104)
    at org.elasticsearch.common.unit.ByteSizeValue.<clinit>(ByteSizeValue
.java:39)
    at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(
BulkProcessor.java:88)
    at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(
BulkProcessor.java:80)
    at org.elasticsearch.action.bulk.BulkProcessor.builder(BulkProcessor
.java:174)

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [Kubernetes] java.lang.OutOfMemoryError: Metaspace ?

Posted by Niels Basjes <Ni...@basjes.nl>.
I've put some information about my situation in the ticket
https://issues.apache.org/jira/browse/FLINK-16142?focusedCommentId=17049679&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17049679

On Mon, Mar 2, 2020 at 2:55 PM Arvid Heise <ar...@ververica.com> wrote:

> Hi Niels,
>
> to add to Yang. 96m is plenty of space and was heavily tested by Alibaba.
>
> The most likely reason and the motivation for the change is that you
> probably have a classloader leak in your pipeline, quite possibly by one of
> our connectors. For example, see FLINK-16142 [1].
> If you could give us more details about your pipeline, we can try to ping
> it down.
>
> However, if the error already occurs during the first start on the
> cluster, then the limit is too narrow and increasing it will already help.
> In that case, please leave us a quick comment with your used dependencies,
> so that we could potentially increase the default setting for future
> releases.
>
> [1] https://issues.apache.org/jira/browse/FLINK-16142
>
> On Mon, Mar 2, 2020 at 1:27 PM Yang Wang <da...@gmail.com> wrote:
>
>> From 1.10, Flink will enable the metaspace limit via
>> "-XX:MaxMetaspaceSize"
>> by default. The default value is 96m, loading too many classes will cause
>> "OutOfMemoryError: Metaspace"[1]. So you need to increase the configured
>> value.
>>
>>
>> [1].
>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_trouble.html#outofmemoryerror-metaspace
>>
>>
>> Best,
>> Yang
>>
>> Niels Basjes <Ni...@basjes.nl> 于2020年3月2日周一 下午7:16写道：
>>
>>> Hi,
>>>
>>> I'm running a lot of batch jobs on Kubernetes once in a while I get this
>>> exception.
>>> What is causing this?
>>> How can I fix this?
>>>
>>> Niels Basjes
>>>
>>> java.lang.OutOfMemoryError: Metaspace
>>>     at java.lang.ClassLoader.defineClass1(Native Method)
>>>     at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
>>>     at java.security.SecureClassLoader.defineClass(SecureClassLoader
>>> .java:142)
>>>     at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
>>>     at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
>>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
>>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
>>>     at org.apache.flink.util.ChildFirstClassLoader.loadClass(
>>> ChildFirstClassLoader.java:60)
>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
>>>     at java.lang.ClassLoader.defineClass1(Native Method)
>>>     at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
>>>     at java.security.SecureClassLoader.defineClass(SecureClassLoader
>>> .java:142)
>>>     at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
>>>     at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
>>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
>>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
>>>     at org.apache.flink.util.ChildFirstClassLoader.loadClass(
>>> ChildFirstClassLoader.java:60)
>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
>>>     at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:60)
>>>     at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(
>>> ESLoggerFactory.java:45)
>>>     at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(
>>> ESLoggerFactory.java:53)
>>>     at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
>>> 104)
>>>     at org.elasticsearch.common.unit.ByteSizeValue.<clinit>(
>>> ByteSizeValue.java:39)
>>>     at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(
>>> BulkProcessor.java:88)
>>>     at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(
>>> BulkProcessor.java:80)
>>>     at org.elasticsearch.action.bulk.BulkProcessor.builder(BulkProcessor
>>> .java:174)
>>>
>>> --
>>> Best regards / Met vriendelijke groeten,
>>>
>>> Niels Basjes
>>>
>>

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [Kubernetes] java.lang.OutOfMemoryError: Metaspace ?

Posted by Arvid Heise <ar...@ververica.com>.
Hi Niels,

to add to Yang. 96m is plenty of space and was heavily tested by Alibaba.

The most likely reason and the motivation for the change is that you
probably have a classloader leak in your pipeline, quite possibly by one of
our connectors. For example, see FLINK-16142 [1].
If you could give us more details about your pipeline, we can try to ping
it down.

However, if the error already occurs during the first start on the cluster,
then the limit is too narrow and increasing it will already help. In that
case, please leave us a quick comment with your used dependencies, so that
we could potentially increase the default setting for future releases.

[1] https://issues.apache.org/jira/browse/FLINK-16142

On Mon, Mar 2, 2020 at 1:27 PM Yang Wang <da...@gmail.com> wrote:

> From 1.10, Flink will enable the metaspace limit via
> "-XX:MaxMetaspaceSize"
> by default. The default value is 96m, loading too many classes will cause
> "OutOfMemoryError: Metaspace"[1]. So you need to increase the configured
> value.
>
>
> [1].
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_trouble.html#outofmemoryerror-metaspace
>
>
> Best,
> Yang
>
> Niels Basjes <Ni...@basjes.nl> 于2020年3月2日周一 下午7:16写道：
>
>> Hi,
>>
>> I'm running a lot of batch jobs on Kubernetes once in a while I get this
>> exception.
>> What is causing this?
>> How can I fix this?
>>
>> Niels Basjes
>>
>> java.lang.OutOfMemoryError: Metaspace
>>     at java.lang.ClassLoader.defineClass1(Native Method)
>>     at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
>>     at java.security.SecureClassLoader.defineClass(SecureClassLoader
>> .java:142)
>>     at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
>>     at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
>>     at org.apache.flink.util.ChildFirstClassLoader.loadClass(
>> ChildFirstClassLoader.java:60)
>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
>>     at java.lang.ClassLoader.defineClass1(Native Method)
>>     at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
>>     at java.security.SecureClassLoader.defineClass(SecureClassLoader
>> .java:142)
>>     at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
>>     at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
>>     at org.apache.flink.util.ChildFirstClassLoader.loadClass(
>> ChildFirstClassLoader.java:60)
>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
>>     at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:60)
>>     at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(
>> ESLoggerFactory.java:45)
>>     at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(
>> ESLoggerFactory.java:53)
>>     at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
>> 104)
>>     at org.elasticsearch.common.unit.ByteSizeValue.<clinit>(ByteSizeValue
>> .java:39)
>>     at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(
>> BulkProcessor.java:88)
>>     at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(
>> BulkProcessor.java:80)
>>     at org.elasticsearch.action.bulk.BulkProcessor.builder(BulkProcessor
>> .java:174)
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>>
>

Re: [Kubernetes] java.lang.OutOfMemoryError: Metaspace ?

Posted by Yang Wang <da...@gmail.com>.
From 1.10, Flink will enable the metaspace limit via "-XX:MaxMetaspaceSize"
by default. The default value is 96m, loading too many classes will cause
"OutOfMemoryError: Metaspace"[1]. So you need to increase the configured
value.


[1].
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_trouble.html#outofmemoryerror-metaspace


Best,
Yang

Niels Basjes <Ni...@basjes.nl> 于2020年3月2日周一 下午7:16写道：

> Hi,
>
> I'm running a lot of batch jobs on Kubernetes once in a while I get this
> exception.
> What is causing this?
> How can I fix this?
>
> Niels Basjes
>
> java.lang.OutOfMemoryError: Metaspace
>     at java.lang.ClassLoader.defineClass1(Native Method)
>     at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
>     at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:
> 142)
>     at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
>     at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
>     at org.apache.flink.util.ChildFirstClassLoader.loadClass(
> ChildFirstClassLoader.java:60)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
>     at java.lang.ClassLoader.defineClass1(Native Method)
>     at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
>     at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:
> 142)
>     at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
>     at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
>     at org.apache.flink.util.ChildFirstClassLoader.loadClass(
> ChildFirstClassLoader.java:60)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
>     at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:60)
>     at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(
> ESLoggerFactory.java:45)
>     at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(
> ESLoggerFactory.java:53)
>     at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:104
> )
>     at org.elasticsearch.common.unit.ByteSizeValue.<clinit>(ByteSizeValue
> .java:39)
>     at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(
> BulkProcessor.java:88)
>     at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(
> BulkProcessor.java:80)
>     at org.elasticsearch.action.bulk.BulkProcessor.builder(BulkProcessor
> .java:174)
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>