You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Alexander Smirnov <al...@gmail.com> on 2018/04/02 11:32:48 UTC

Kafka exceptions in Flink log file

I see a lot of messages in flink log like below. What's the cause?


02 Apr 2018 04:09:13,554 ERROR
org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
kafka producer I/O thread:
org.apache.kafka.common.KafkaException: Error registering mbean
kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
        at
org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:163)
        at
org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:81)
        at
org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:504)
        at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:255)
        at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:240)
        at
org.apache.kafka.common.network.Selector$SelectorMetrics.maybeRegisterConnectionMetrics(Selector.java:811)
        at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:353)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
        at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
        at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
        at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.management.InstanceAlreadyExistsException:
kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
        at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
        at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
        at
org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:161)
        ... 11 more
02 Apr 2018 04:09:13,673 ERROR org.apache.kafka.common.utils.KafkaThread -
Uncaught exception in kafka-producer-network-thread | producer-3:
java.lang.NoClassDefFoundError: org/apache/kafka/clients/NetworkClient$1
        at
org.apache.kafka.clients.NetworkClient.processDisconnection(NetworkClient.java:583)
        at
org.apache.kafka.clients.NetworkClient.handleDisconnections(NetworkClient.java:705)
        at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:443)
        at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
        at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.apache.kafka.clients.NetworkClient$1
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at
org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 6 more


Thank you,
Alex

Re: Kafka exceptions in Flink log file

Posted by Alexander Smirnov <al...@gmail.com>.

Hi Timo,

it is the latest released version - 1.4.2

This only happens when a job falls into a restart loop and stays in it for
20 minutes or so.
Looks like for each restart, Flink loads classes anew, but previously
loaded classes are not garbage collected for some reason (still referenced?)

Very soon, JVM runs out of Metaspace and this error occurs.

Thank you,
Alex



On Tue, Apr 3, 2018 at 4:24 PM Timo Walther <tw...@apache.org> wrote:

> Hi Alex,
>
> which version of Flink are you running? There were some class loading
> issues with Kafka recently. I would try it with the newest Flink
> version. Otherwise ClassNotFoundException usually indicates that
> something is wrong with your dependencies. Maybe you can share your
> pom.xml with us.
>
> Regards,
> Timo
>
> Am 02.04.18 um 13:32 schrieb Alexander Smirnov:
> > I see a lot of messages in flink log like below. What's the cause?
> >
> >
> > 02 Apr 2018 04:09:13,554 ERROR
> > org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
> > kafka producer I/O thread:
> > org.apache.kafka.common.KafkaException: Error registering mbean
> >
> kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
> >         at
> >
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:163)
> >         at
> >
> org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:81)
> >         at
> > org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:504)
> >         at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:255)
> >         at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:240)
> >         at
> >
> org.apache.kafka.common.network.Selector$SelectorMetrics.maybeRegisterConnectionMetrics(Selector.java:811)
> >         at
> >
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:353)
> >         at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:326)
> >         at
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
> >         at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
> >         at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
> >         at java.lang.Thread.run(Thread.java:748)
> > Caused by: javax.management.InstanceAlreadyExistsException:
> >
> kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
> >         at
> > com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> >         at
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> >         at
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> >         at
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> >         at
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> >         at
> >
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> >         at
> >
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:161)
> >         ... 11 more
> > 02 Apr 2018 04:09:13,673 ERROR
> > org.apache.kafka.common.utils.KafkaThread - Uncaught exception in
> > kafka-producer-network-thread | producer-3:
> > java.lang.NoClassDefFoundError: org/apache/kafka/clients/NetworkClient$1
> >         at
> >
> org.apache.kafka.clients.NetworkClient.processDisconnection(NetworkClient.java:583)
> >         at
> >
> org.apache.kafka.clients.NetworkClient.handleDisconnections(NetworkClient.java:705)
> >         at
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:443)
> >         at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
> >         at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
> >         at java.lang.Thread.run(Thread.java:748)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.kafka.clients.NetworkClient$1
> >         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >         at
> >
> org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >         ... 6 more
> >
> >
> > Thank you,
> > Alex
>
>
>

Re: Kafka exceptions in Flink log file

Posted by Timo Walther <tw...@apache.org>.

Hi Alex,

which version of Flink are you running? There were some class loading 
issues with Kafka recently. I would try it with the newest Flink 
version. Otherwise ClassNotFoundException usually indicates that 
something is wrong with your dependencies. Maybe you can share your 
pom.xml with us.

Regards,
Timo

Am 02.04.18 um 13:32 schrieb Alexander Smirnov:
> I see a lot of messages in flink log like below. What's the cause?
>
>
> 02 Apr 2018 04:09:13,554 ERROR 
> org.apache.kafka.clients.producer.internals.Sender - Uncaught error in 
> kafka producer I/O thread:
> org.apache.kafka.common.KafkaException: Error registering mbean 
> kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
>         at 
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:163)
>         at 
> org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:81)
>         at 
> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:504)
>         at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:255)
>         at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:240)
>         at 
> org.apache.kafka.common.network.Selector$SelectorMetrics.maybeRegisterConnectionMetrics(Selector.java:811)
>         at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:353)
>         at 
> org.apache.kafka.common.network.Selector.poll(Selector.java:326)
>         at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
>         at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
>         at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.management.InstanceAlreadyExistsException: 
> kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
>         at 
> com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
>         at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
>         at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
>         at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
>         at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
>         at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
>         at 
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:161)
>         ... 11 more
> 02 Apr 2018 04:09:13,673 ERROR 
> org.apache.kafka.common.utils.KafkaThread - Uncaught exception in 
> kafka-producer-network-thread | producer-3:
> java.lang.NoClassDefFoundError: org/apache/kafka/clients/NetworkClient$1
>         at 
> org.apache.kafka.clients.NetworkClient.processDisconnection(NetworkClient.java:583)
>         at 
> org.apache.kafka.clients.NetworkClient.handleDisconnections(NetworkClient.java:705)
>         at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:443)
>         at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
>         at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.kafka.clients.NetworkClient$1
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at 
> org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 6 more
>
>
> Thank you,
> Alex