You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nanda Naga <na...@microsoft.com.INVALID> on 2025/05/29 16:17:30 UTC
RE: [EXTERNAL] Re: What config to avoid outofmemory exceptions happening in kafka server logs
Thanks Radu for the response.
Wondering why it is out of memory? Any config to resolve this or any change in client to resolve this?
Also, we use librdkafka client to connect to plaintext port. The memory errors seems intermittent as I see produces happening to plaintext as well.
Regards,
Nanda
-----Original Message-----
From: Radu Radutiu <rr...@gmail.com>
Sent: Wednesday, May 28, 2025 3:08 AM
To: users@kafka.apache.org
Subject: [EXTERNAL] Re: What config to avoid outofmemory exceptions happening in kafka server logs
[You don't often get email from rradutiu@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
The error usually occurs when you try to connect with a client speaking TLS to the plain text port of the Kafka broker.
Radu
On Wed, May 28, 2025 at 12:37 AM Nanda Naga <na...@microsoft.com.invalid>
wrote:
> I see this exception in our environment in multiple machines. What
> config will help to resolve this error?
>
> The exception I get is at
>
> java.lang.OutOfMemoryError: Unable to allocate 369295624 bytes at
> java.base/jdk.internal.misc.Unsafe.allocateMemory(Unsafe.java:632)
> at
> java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:125)
> at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332)
> at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:228)
> at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:293)
> at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:259)
> at
> java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:417
> )
> at
> org.apache.kafka.common.network.PlaintextTransportLayer.read(Plaintext
> TransportLayer.java:103)
> at
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive
> .java:118)
> at
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java
> :452) at
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:40
> 2) at
> org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674
> )
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.ja
> va:576) at
> org.apache.kafka.common.network.Selector.poll(Selector.java:481)
> at kafka.network.Processor.poll(SocketServer.scala:1144)
> at kafka.network.Processor.run(SocketServer.scala:1047)
> at java.base/java.lang.Thread.run(Thread.java:840)
>
> Regards,
> Nanda
>
>
RE: [EXTERNAL] Re: What config to avoid outofmemory exceptions happening in kafka server logs
Posted by Nanda Naga <na...@microsoft.com.INVALID>.
Thanks Omer for answer. It helps to understand. Couple of follow up questions
1. Is there a way to confirm this using any kafka logs (message with encrypted info).
2. Is this error intermittent or happens always? Because I do see plaintext calls succeed for the same client and topic
Regards,
Nanda
-----Original Message-----
From: Ömer Şiar Baysal <os...@gmail.com>
Sent: Thursday, May 29, 2025 1:19 PM
To: users@kafka.apache.org
Cc: rradutiu@gmail.com
Subject: Re: [EXTERNAL] Re: What config to avoid outofmemory exceptions happening in kafka server logs
Hi,
This is a known issue,
https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-4493
Basically when broker receives first few bytes of the encrypted message on its plaintext listener, it may end up trying to allocate a huge buffer for the message, because it assumes the received bytes are for plaintext listener. Since encrypted message is no longer a clear text, its first 4 bytes can resemble anything (sometimes a huge number) thus resulting in OOM.
Hope that clears it up for you.
Ömer Şiar Baysal
On Thu, May 29, 2025, 18:18 Nanda Naga <na...@microsoft.com.invalid>
wrote:
> Thanks Radu for the response.
>
> Wondering why it is out of memory? Any config to resolve this or any
> change in client to resolve this?
>
> Also, we use librdkafka client to connect to plaintext port. The
> memory errors seems intermittent as I see produces happening to plaintext as well.
>
> Regards,
> Nanda
>
> -----Original Message-----
> From: Radu Radutiu <rr...@gmail.com>
> Sent: Wednesday, May 28, 2025 3:08 AM
> To: users@kafka.apache.org
> Subject: [EXTERNAL] Re: What config to avoid outofmemory exceptions
> happening in kafka server logs
>
> [You don't often get email from rradutiu@gmail.com. Learn why this is
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> The error usually occurs when you try to connect with a client
> speaking TLS to the plain text port of the Kafka broker.
>
> Radu
>
> On Wed, May 28, 2025 at 12:37 AM Nanda Naga <nandanaga@microsoft.com
> .invalid>
> wrote:
>
> > I see this exception in our environment in multiple machines. What
> > config will help to resolve this error?
> >
> > The exception I get is at
> >
> > java.lang.OutOfMemoryError: Unable to allocate 369295624 bytes at
> > java.base/jdk.internal.misc.Unsafe.allocateMemory(Unsafe.java:632)
> > at
> > java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:125
> > ) at
> > java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332)
> > at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:228)
> > at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:293)
> > at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:259)
> > at
> > java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:4
> > 17
> > )
> > at
> > org.apache.kafka.common.network.PlaintextTransportLayer.read(Plainte
> > xt
> > TransportLayer.java:103)
> > at
> > org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkRecei
> > ve
> > .java:118)
> > at
> > org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.ja
> > va
> > :452) at
> > org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:
> > 40
> > 2) at
> > org.apache.kafka.common.network.Selector.attemptRead(Selector.java:6
> > 74
> > )
> > at
> > org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.
> > ja
> > va:576) at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:481)
> > at kafka.network.Processor.poll(SocketServer.scala:1144)
> > at kafka.network.Processor.run(SocketServer.scala:1047)
> > at java.base/java.lang.Thread.run(Thread.java:840)
> >
> > Regards,
> > Nanda
> >
> >
>
Re: [EXTERNAL] Re: What config to avoid outofmemory exceptions happening in kafka server logs
Posted by Ömer Şiar Baysal <os...@gmail.com>.
Hi,
This is a known issue,
https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-4493
Basically when broker receives first few bytes of the encrypted message on
its plaintext listener, it may end up trying to allocate a huge buffer for
the message, because it assumes the received bytes are for plaintext
listener. Since encrypted message is no longer a clear text, its first 4
bytes can resemble anything (sometimes a huge number) thus resulting in OOM.
Hope that clears it up for you.
Ömer Şiar Baysal
On Thu, May 29, 2025, 18:18 Nanda Naga <na...@microsoft.com.invalid>
wrote:
> Thanks Radu for the response.
>
> Wondering why it is out of memory? Any config to resolve this or any
> change in client to resolve this?
>
> Also, we use librdkafka client to connect to plaintext port. The memory
> errors seems intermittent as I see produces happening to plaintext as well.
>
> Regards,
> Nanda
>
> -----Original Message-----
> From: Radu Radutiu <rr...@gmail.com>
> Sent: Wednesday, May 28, 2025 3:08 AM
> To: users@kafka.apache.org
> Subject: [EXTERNAL] Re: What config to avoid outofmemory exceptions
> happening in kafka server logs
>
> [You don't often get email from rradutiu@gmail.com. Learn why this is
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> The error usually occurs when you try to connect with a client speaking
> TLS to the plain text port of the Kafka broker.
>
> Radu
>
> On Wed, May 28, 2025 at 12:37 AM Nanda Naga <nandanaga@microsoft.com
> .invalid>
> wrote:
>
> > I see this exception in our environment in multiple machines. What
> > config will help to resolve this error?
> >
> > The exception I get is at
> >
> > java.lang.OutOfMemoryError: Unable to allocate 369295624 bytes at
> > java.base/jdk.internal.misc.Unsafe.allocateMemory(Unsafe.java:632)
> > at
> > java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:125)
> > at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332)
> > at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:228)
> > at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:293)
> > at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:259)
> > at
> > java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:417
> > )
> > at
> > org.apache.kafka.common.network.PlaintextTransportLayer.read(Plaintext
> > TransportLayer.java:103)
> > at
> > org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive
> > .java:118)
> > at
> > org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java
> > :452) at
> > org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:40
> > 2) at
> > org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674
> > )
> > at
> > org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.ja
> > va:576) at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:481)
> > at kafka.network.Processor.poll(SocketServer.scala:1144)
> > at kafka.network.Processor.run(SocketServer.scala:1047)
> > at java.base/java.lang.Thread.run(Thread.java:840)
> >
> > Regards,
> > Nanda
> >
> >
>