rpc invocation exceeds the maximum akka framesize

["Deshpande, Omkar" <Om...@intuit.com>]

Hello,

We run a lot of flink applications. Some of them sometimes run into this error on Job Manager-
The rpc invocation size exceeds the maximum akka framesize

After we increase the framesize the application starts working again.
What factors determine the akka framesize? We sometimes see applications run without this issue for months and then run into this error.
How can we determine the framesize before running into this error?

Thanks,
Omkar

Re: rpc invocation exceeds the maximum akka framesize

[Roman Khachatryan <ro...@apache.org>]

Hi,

There could be many reasons for exceeding akka framesize, for example
1. "inlined" state that is stored in checkpoint .metadata file (rather
than "data" files - see [1])
2. broadcast state as you mentioned (though only the metadata is sent
unless the data fits the above limits)
3. too many state handles in case of aggressive downscaling
4. too many upstream/downstream tasks in case of high parallelism (as
the task receives the needed information about them)
5. the task reports too many metrics or other details (e.g. exception
stacktrace or maybe flamegraph)

[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/config/#state-storage-fs-memory-threshold

Regards,
Roman

On Tue, Sep 28, 2021 at 3:32 AM Deshpande, Omkar
<Om...@intuit.com> wrote:
>
> Hello,
>
> We run a lot of flink applications. Some of them sometimes run into this error on Job Manager-
> The rpc invocation size exceeds the maximum akka framesize
>
> After we increase the framesize the application starts working again.
> What factors determine the akka framesize? We sometimes see applications run without this issue for months and then run into this error.
> How can we determine the framesize before running into this error?
>
> Thanks,
> Omkar

Re: rpc invocation exceeds the maximum akka framesize

["Deshpande, Omkar" <Om...@intuit.com>]

Is the akka framesize a function of broadcast size?
________________________________
From: Deshpande, Omkar
Sent: Monday, September 27, 2021 6:31 PM
To: user@flink.apache.org <us...@flink.apache.org>
Cc: Benenson, Mikhail <Mi...@intuit.com>; Hwang, Nick <Ni...@intuit.com>; Canchi, Srivathsan <SR...@intuit.com>
Subject: rpc invocation exceeds the maximum akka framesize

Hello,

We run a lot of flink applications. Some of them sometimes run into this error on Job Manager-
The rpc invocation size exceeds the maximum akka framesize

After we increase the framesize the application starts working again.
What factors determine the akka framesize? We sometimes see applications run without this issue for months and then run into this error.
How can we determine the framesize before running into this error?

Thanks,
Omkar