different behaviour for OOM error in Flink 1.16

[neha goyal <ne...@gmail.com>]

Hello,
I want to understand the reason behind the different behavior of Flink
pipelines. When I am running a Kafka in and delta out pipeline with fewer
resources than needed for a particular pipeline, It fails with OOM error.
In the newer version, it takes around 8 minutes to give the first exception
while in the older version, it fails in 2 minutes only. what is the new
change which can explain this behavior? we have enabled the retrials, hence
using the 'first' term.

Pipeline1
Flink version: 1.16.1
Time taken for the first OOM error: 8 mins

Pipeline2
Flink version: 1.13.6
Time taken for the first OOM error: 2 mins
Attaching the screenshot of the error.

Re: different behaviour for OOM error in Flink 1.16

[Biao Geng <bi...@gmail.com>]

Hi,
There are plenty of code changes comparing 1.13.6 with 1.16.1. The diff can
be caused by the runtime module, connector module and many other reasons.
So it is hard to give a direct answer. But if you like, you can dump the
memory of the job in different versions.
Then, use tools like MAT, you can find some insights and then go to check
relevant codes.

Best,
Biao Geng

neha goyal <ne...@gmail.com> 于2023年4月21日周五 18:42写道：

> Hello,
> I want to understand the reason behind the different behavior of Flink
> pipelines. When I am running a Kafka in and delta out pipeline with fewer
> resources than needed for a particular pipeline, It fails with OOM error.
> In the newer version, it takes around 8 minutes to give the first exception
> while in the older version, it fails in 2 minutes only. what is the new
> change which can explain this behavior? we have enabled the retrials, hence
> using the 'first' term.
>
> Pipeline1
> Flink version: 1.16.1
> Time taken for the first OOM error: 8 mins
>
> Pipeline2
> Flink version: 1.13.6
> Time taken for the first OOM error: 2 mins
> Attaching the screenshot of the error.
>