You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Ori Popowski <or...@gmail.com> on 2022/03/15 05:52:25 UTC
Adjusted frame length exceeds 2147483647
I am running a production job for at least 1 year, and I got to day this
error:
org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
Adjusted frame length exceeds 2147483647: 2969686273 - discarded
(connection to
'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
10.208.65.38:40737')
Nothing was changed in the code for a long time. What's causing this error
and how to fix it? I am running Flink 1.10.3 on YARN.
This is the full stack trace:
2022-03-15 03:22:13
org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
Adjusted frame length exceeds 2147483647: 2969686273 - discarded
(connection to
'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
10.208.65.38:40737')
at org.apache.flink.runtime.io.network.netty.
CreditBasedPartitionRequestClientHandler.exceptionCaught(
CreditBasedPartitionRequestClientHandler.java:165)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeExceptionCaught(
AbstractChannelHandlerContext.java:297)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeExceptionCaught(
AbstractChannelHandlerContext.java:276)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.fireExceptionCaught(
AbstractChannelHandlerContext.java:268)
at org.apache.flink.shaded.netty4.io.netty.channel.
ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter
.java:143)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeExceptionCaught(
AbstractChannelHandlerContext.java:297)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.notifyHandlerException(
AbstractChannelHandlerContext.java:831)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeChannelRead(
AbstractChannelHandlerContext.java:376)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeChannelRead(
AbstractChannelHandlerContext.java:360)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
.java:352)
at org.apache.flink.shaded.netty4.io.netty.channel.
DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:
1421)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeChannelRead(
AbstractChannelHandlerContext.java:374)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeChannelRead(
AbstractChannelHandlerContext.java:360)
at org.apache.flink.shaded.netty4.io.netty.channel.
DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.
AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
.processSelectedKey(NioEventLoop.java:697)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
.processSelectedKeysOptimized(NioEventLoop.java:632)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
.processSelectedKeys(NioEventLoop.java:549)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(
NioEventLoop.java:511)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at org.apache.flink.shaded.netty4.io.netty.util.internal.
ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.shaded.netty4.io.netty.handler.codec.
TooLongFrameException: Adjusted frame length exceeds 2147483647: 2969686273
- discarded
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder
.java:491)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
LengthFieldBasedFrameDecoder.exceededFrameLength(
LengthFieldBasedFrameDecoder.java:378)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:421)
at org.apache.flink.runtime.io.network.netty.
NettyMessage$NettyMessageDecoder.decode(NettyMessage.java:214)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:334)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder
.java:505)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeChannelRead(
AbstractChannelHandlerContext.java:374)
... 14 more
Re: Adjusted frame length exceeds 2147483647
Posted by Ori Popowski <or...@gmail.com>.
I am not aware that we use Nessus. We are running on Google Cloud Dataproc
vanilla. On the other hand, Flink restarted from a checkpoint and reread
the Kafka offset and didn't encounter this error again, so if it was
something in the input I would expect it to happen again when it processes
the Kafka record that caused the problem.
If it happens again I will enable TLS as described in the ticket.
On Fri, Mar 18, 2022 at 8:57 AM Matthias Pohl <ma...@apache.org> wrote:
> One other pointer: Martijn mentioned in FLINK-24923 [1] tools like Nessus
> could generate traffic while scanning for ports. It's just the size of the
> request that is suspicious.
>
> [1] https://issues.apache.org/jira/browse/FLINK-24923
>
> On Thu, Mar 17, 2022 at 5:29 PM Ori Popowski <or...@gmail.com> wrote:
>
>> This issue did not repeat, so it may be a network issue
>>
>> On Thu, Mar 17, 2022 at 6:12 PM Matthias Pohl <ma...@apache.org> wrote:
>>
>>> Hi Ori,
>>> that looks odd. The message seems to exceed the maximum size
>>> of 2147483647 bytes (2GB). I couldn't find anything similar in the ML or in
>>> Jira that supports a bug in Flink. Could it be that there was some network
>>> issue?
>>>
>>> Matthias
>>>
>>> On Tue, Mar 15, 2022 at 6:52 AM Ori Popowski <or...@gmail.com> wrote:
>>>
>>>> I am running a production job for at least 1 year, and I got to day
>>>> this error:
>>>>
>>>>
>>>> org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
>>>> Adjusted frame length exceeds 2147483647: 2969686273 - discarded
>>>> (connection to
>>>> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
>>>> 10.208.65.38:40737')
>>>>
>>>> Nothing was changed in the code for a long time. What's causing this
>>>> error and how to fix it? I am running Flink 1.10.3 on YARN.
>>>>
>>>> This is the full stack trace:
>>>>
>>>> 2022-03-15 03:22:13
>>>> org.apache.flink.runtime.io.network.netty.exception.
>>>> LocalTransportException: Adjusted frame length exceeds 2147483647:
>>>> 2969686273 - discarded (connection to
>>>> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
>>>> 10.208.65.38:40737')
>>>> at org.apache.flink.runtime.io.network.netty.
>>>> CreditBasedPartitionRequestClientHandler.exceptionCaught(
>>>> CreditBasedPartitionRequestClientHandler.java:165)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> AbstractChannelHandlerContext.invokeExceptionCaught(
>>>> AbstractChannelHandlerContext.java:297)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> AbstractChannelHandlerContext.invokeExceptionCaught(
>>>> AbstractChannelHandlerContext.java:276)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> AbstractChannelHandlerContext.fireExceptionCaught(
>>>> AbstractChannelHandlerContext.java:268)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> ChannelInboundHandlerAdapter.exceptionCaught(
>>>> ChannelInboundHandlerAdapter.java:143)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> AbstractChannelHandlerContext.invokeExceptionCaught(
>>>> AbstractChannelHandlerContext.java:297)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> AbstractChannelHandlerContext.notifyHandlerException(
>>>> AbstractChannelHandlerContext.java:831)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> AbstractChannelHandlerContext.invokeChannelRead(
>>>> AbstractChannelHandlerContext.java:376)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> AbstractChannelHandlerContext.invokeChannelRead(
>>>> AbstractChannelHandlerContext.java:360)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> AbstractChannelHandlerContext.fireChannelRead(
>>>> AbstractChannelHandlerContext.java:352)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline
>>>> .java:1421)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> AbstractChannelHandlerContext.invokeChannelRead(
>>>> AbstractChannelHandlerContext.java:374)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> AbstractChannelHandlerContext.invokeChannelRead(
>>>> AbstractChannelHandlerContext.java:360)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.
>>>> AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:
>>>> 163)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>>>> .processSelectedKey(NioEventLoop.java:697)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>>>> .processSelectedKeysOptimized(NioEventLoop.java:632)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>>>> .processSelectedKeys(NioEventLoop.java:549)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>>>> .run(NioEventLoop.java:511)
>>>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
>>>> SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
>>>> at org.apache.flink.shaded.netty4.io.netty.util.internal.
>>>> ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>>> TooLongFrameException: Adjusted frame length exceeds 2147483647:
>>>> 2969686273 - discarded
>>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>>> LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513
>>>> )
>>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>>> LengthFieldBasedFrameDecoder.failIfNecessary(
>>>> LengthFieldBasedFrameDecoder.java:491)
>>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>>> LengthFieldBasedFrameDecoder.exceededFrameLength(
>>>> LengthFieldBasedFrameDecoder.java:378)
>>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>>> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:
>>>> 421)
>>>> at org.apache.flink.runtime.io.network.netty.
>>>> NettyMessage$NettyMessageDecoder.decode(NettyMessage.java:214)
>>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>>> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:
>>>> 334)
>>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>>> ByteToMessageDecoder.decodeRemovalReentryProtection(
>>>> ByteToMessageDecoder.java:505)
>>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>>> ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
>>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>>> ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
>>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>>> AbstractChannelHandlerContext.invokeChannelRead(
>>>> AbstractChannelHandlerContext.java:374)
>>>> ... 14 more
>>>>
>>>>
>>>>
Re: Adjusted frame length exceeds 2147483647
Posted by Matthias Pohl <ma...@apache.org>.
One other pointer: Martijn mentioned in FLINK-24923 [1] tools like Nessus
could generate traffic while scanning for ports. It's just the size of the
request that is suspicious.
[1] https://issues.apache.org/jira/browse/FLINK-24923
On Thu, Mar 17, 2022 at 5:29 PM Ori Popowski <or...@gmail.com> wrote:
> This issue did not repeat, so it may be a network issue
>
> On Thu, Mar 17, 2022 at 6:12 PM Matthias Pohl <ma...@apache.org> wrote:
>
>> Hi Ori,
>> that looks odd. The message seems to exceed the maximum size
>> of 2147483647 bytes (2GB). I couldn't find anything similar in the ML or in
>> Jira that supports a bug in Flink. Could it be that there was some network
>> issue?
>>
>> Matthias
>>
>> On Tue, Mar 15, 2022 at 6:52 AM Ori Popowski <or...@gmail.com> wrote:
>>
>>> I am running a production job for at least 1 year, and I got to day this
>>> error:
>>>
>>>
>>> org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
>>> Adjusted frame length exceeds 2147483647: 2969686273 - discarded
>>> (connection to
>>> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
>>> 10.208.65.38:40737')
>>>
>>> Nothing was changed in the code for a long time. What's causing this
>>> error and how to fix it? I am running Flink 1.10.3 on YARN.
>>>
>>> This is the full stack trace:
>>>
>>> 2022-03-15 03:22:13
>>> org.apache.flink.runtime.io.network.netty.exception.
>>> LocalTransportException: Adjusted frame length exceeds 2147483647:
>>> 2969686273 - discarded (connection to
>>> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
>>> 10.208.65.38:40737')
>>> at org.apache.flink.runtime.io.network.netty.
>>> CreditBasedPartitionRequestClientHandler.exceptionCaught(
>>> CreditBasedPartitionRequestClientHandler.java:165)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> AbstractChannelHandlerContext.invokeExceptionCaught(
>>> AbstractChannelHandlerContext.java:297)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> AbstractChannelHandlerContext.invokeExceptionCaught(
>>> AbstractChannelHandlerContext.java:276)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> AbstractChannelHandlerContext.fireExceptionCaught(
>>> AbstractChannelHandlerContext.java:268)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> ChannelInboundHandlerAdapter.exceptionCaught(
>>> ChannelInboundHandlerAdapter.java:143)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> AbstractChannelHandlerContext.invokeExceptionCaught(
>>> AbstractChannelHandlerContext.java:297)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> AbstractChannelHandlerContext.notifyHandlerException(
>>> AbstractChannelHandlerContext.java:831)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> AbstractChannelHandlerContext.invokeChannelRead(
>>> AbstractChannelHandlerContext.java:376)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> AbstractChannelHandlerContext.invokeChannelRead(
>>> AbstractChannelHandlerContext.java:360)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> AbstractChannelHandlerContext.fireChannelRead(
>>> AbstractChannelHandlerContext.java:352)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline
>>> .java:1421)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> AbstractChannelHandlerContext.invokeChannelRead(
>>> AbstractChannelHandlerContext.java:374)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> AbstractChannelHandlerContext.invokeChannelRead(
>>> AbstractChannelHandlerContext.java:360)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.
>>> AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:
>>> 163)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>>> .processSelectedKey(NioEventLoop.java:697)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>>> .processSelectedKeysOptimized(NioEventLoop.java:632)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>>> .processSelectedKeys(NioEventLoop.java:549)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>>> .run(NioEventLoop.java:511)
>>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
>>> SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
>>> at org.apache.flink.shaded.netty4.io.netty.util.internal.
>>> ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>> TooLongFrameException: Adjusted frame length exceeds 2147483647:
>>> 2969686273 - discarded
>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>> LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513)
>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>> LengthFieldBasedFrameDecoder.failIfNecessary(
>>> LengthFieldBasedFrameDecoder.java:491)
>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>> LengthFieldBasedFrameDecoder.exceededFrameLength(
>>> LengthFieldBasedFrameDecoder.java:378)
>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:
>>> 421)
>>> at org.apache.flink.runtime.io.network.netty.
>>> NettyMessage$NettyMessageDecoder.decode(NettyMessage.java:214)
>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:
>>> 334)
>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>> ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder
>>> .java:505)
>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>> ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
>>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>> ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
>>> at org.apache.flink.shaded.netty4.io.netty.channel.
>>> AbstractChannelHandlerContext.invokeChannelRead(
>>> AbstractChannelHandlerContext.java:374)
>>> ... 14 more
>>>
>>>
>>>
Re: Adjusted frame length exceeds 2147483647
Posted by Ori Popowski <or...@gmail.com>.
This issue did not repeat, so it may be a network issue
On Thu, Mar 17, 2022 at 6:12 PM Matthias Pohl <ma...@apache.org> wrote:
> Hi Ori,
> that looks odd. The message seems to exceed the maximum size of 2147483647
> bytes (2GB). I couldn't find anything similar in the ML or in Jira that
> supports a bug in Flink. Could it be that there was some network issue?
>
> Matthias
>
> On Tue, Mar 15, 2022 at 6:52 AM Ori Popowski <or...@gmail.com> wrote:
>
>> I am running a production job for at least 1 year, and I got to day this
>> error:
>>
>>
>> org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
>> Adjusted frame length exceeds 2147483647: 2969686273 - discarded
>> (connection to
>> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
>> 10.208.65.38:40737')
>>
>> Nothing was changed in the code for a long time. What's causing this
>> error and how to fix it? I am running Flink 1.10.3 on YARN.
>>
>> This is the full stack trace:
>>
>> 2022-03-15 03:22:13
>> org.apache.flink.runtime.io.network.netty.exception.
>> LocalTransportException: Adjusted frame length exceeds 2147483647:
>> 2969686273 - discarded (connection to
>> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
>> 10.208.65.38:40737')
>> at org.apache.flink.runtime.io.network.netty.
>> CreditBasedPartitionRequestClientHandler.exceptionCaught(
>> CreditBasedPartitionRequestClientHandler.java:165)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> AbstractChannelHandlerContext.invokeExceptionCaught(
>> AbstractChannelHandlerContext.java:297)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> AbstractChannelHandlerContext.invokeExceptionCaught(
>> AbstractChannelHandlerContext.java:276)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> AbstractChannelHandlerContext.fireExceptionCaught(
>> AbstractChannelHandlerContext.java:268)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter
>> .java:143)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> AbstractChannelHandlerContext.invokeExceptionCaught(
>> AbstractChannelHandlerContext.java:297)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> AbstractChannelHandlerContext.notifyHandlerException(
>> AbstractChannelHandlerContext.java:831)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> AbstractChannelHandlerContext.invokeChannelRead(
>> AbstractChannelHandlerContext.java:376)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> AbstractChannelHandlerContext.invokeChannelRead(
>> AbstractChannelHandlerContext.java:360)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> AbstractChannelHandlerContext.fireChannelRead(
>> AbstractChannelHandlerContext.java:352)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline
>> .java:1421)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> AbstractChannelHandlerContext.invokeChannelRead(
>> AbstractChannelHandlerContext.java:374)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> AbstractChannelHandlerContext.invokeChannelRead(
>> AbstractChannelHandlerContext.java:360)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.
>> AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163
>> )
>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>> .processSelectedKey(NioEventLoop.java:697)
>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>> .processSelectedKeysOptimized(NioEventLoop.java:632)
>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>> .processSelectedKeys(NioEventLoop.java:549)
>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>> .run(NioEventLoop.java:511)
>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
>> SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
>> at org.apache.flink.shaded.netty4.io.netty.util.internal.
>> ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.flink.shaded.netty4.io.netty.handler.codec.
>> TooLongFrameException: Adjusted frame length exceeds 2147483647:
>> 2969686273 - discarded
>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>> LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513)
>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>> LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder
>> .java:491)
>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>> LengthFieldBasedFrameDecoder.exceededFrameLength(
>> LengthFieldBasedFrameDecoder.java:378)
>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:421
>> )
>> at org.apache.flink.runtime.io.network.netty.
>> NettyMessage$NettyMessageDecoder.decode(NettyMessage.java:214)
>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:334
>> )
>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>> ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder
>> .java:505)
>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>> ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
>> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>> ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
>> at org.apache.flink.shaded.netty4.io.netty.channel.
>> AbstractChannelHandlerContext.invokeChannelRead(
>> AbstractChannelHandlerContext.java:374)
>> ... 14 more
>>
>>
>>
Re: Adjusted frame length exceeds 2147483647
Posted by Matthias Pohl <ma...@apache.org>.
Hi Ori,
that looks odd. The message seems to exceed the maximum size of 2147483647
bytes (2GB). I couldn't find anything similar in the ML or in Jira that
supports a bug in Flink. Could it be that there was some network issue?
Matthias
On Tue, Mar 15, 2022 at 6:52 AM Ori Popowski <or...@gmail.com> wrote:
> I am running a production job for at least 1 year, and I got to day this
> error:
>
>
> org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
> Adjusted frame length exceeds 2147483647: 2969686273 - discarded
> (connection to
> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
> 10.208.65.38:40737')
>
> Nothing was changed in the code for a long time. What's causing this error
> and how to fix it? I am running Flink 1.10.3 on YARN.
>
> This is the full stack trace:
>
> 2022-03-15 03:22:13
> org.apache.flink.runtime.io.network.netty.exception.
> LocalTransportException: Adjusted frame length exceeds 2147483647:
> 2969686273 - discarded (connection to
> 'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
> 10.208.65.38:40737')
> at org.apache.flink.runtime.io.network.netty.
> CreditBasedPartitionRequestClientHandler.exceptionCaught(
> CreditBasedPartitionRequestClientHandler.java:165)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeExceptionCaught(
> AbstractChannelHandlerContext.java:297)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeExceptionCaught(
> AbstractChannelHandlerContext.java:276)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.fireExceptionCaught(
> AbstractChannelHandlerContext.java:268)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter
> .java:143)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeExceptionCaught(
> AbstractChannelHandlerContext.java:297)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.notifyHandlerException(
> AbstractChannelHandlerContext.java:831)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeChannelRead(
> AbstractChannelHandlerContext.java:376)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeChannelRead(
> AbstractChannelHandlerContext.java:360)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.fireChannelRead(
> AbstractChannelHandlerContext.java:352)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline
> .java:1421)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeChannelRead(
> AbstractChannelHandlerContext.java:374)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeChannelRead(
> AbstractChannelHandlerContext.java:360)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
> at org.apache.flink.shaded.netty4.io.netty.channel.nio.
> AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
> .processSelectedKey(NioEventLoop.java:697)
> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
> .processSelectedKeysOptimized(NioEventLoop.java:632)
> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
> .processSelectedKeys(NioEventLoop.java:549)
> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
> .run(NioEventLoop.java:511)
> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
> SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
> at org.apache.flink.shaded.netty4.io.netty.util.internal.
> ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.shaded.netty4.io.netty.handler.codec.
> TooLongFrameException: Adjusted frame length exceeds 2147483647:
> 2969686273 - discarded
> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
> LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513)
> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
> LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder
> .java:491)
> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
> LengthFieldBasedFrameDecoder.exceededFrameLength(
> LengthFieldBasedFrameDecoder.java:378)
> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:421)
> at org.apache.flink.runtime.io.network.netty.
> NettyMessage$NettyMessageDecoder.decode(NettyMessage.java:214)
> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
> LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:334)
> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
> ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder
> .java:505)
> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
> ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
> at org.apache.flink.shaded.netty4.io.netty.handler.codec.
> ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
> at org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeChannelRead(
> AbstractChannelHandlerContext.java:374)
> ... 14 more
>
>
>