OOM error behaviour change in newer version

[neha goyal <ne...@gmail.com>]

Hello,
I want to understand the reason behind the different behavior of
Flink-delta pipelines. When I am running a Kafka in and delta out pipeline
with fewer resources than needed for a particular pipeline, It fails with
an OOM error. In the newer version, it takes around 8 minutes to give the
first exception while in the older version, it fails in 2 minutes only.
what is the new change which can explain this behavior? we have enabled the
retrials, hence using the 'first' term.

Pipeline1
Flink version: 1.16.1
Delta connector version: 0.6.0
Time taken for the first OOM error: 8 mins


Pipeline2
Flink version: 1.13.6
Delta connector version: 0.5.0
Time taken for the first OOM error: 2 mins

Re: OOM error behaviour change in newer version

[Shammon FY <zj...@gmail.com>]

Hi neha

I think you can compare the detailed information about memory usage or
error messages. You can also dump heap memory to analyse the objects with
some tools for the two versions of flink

Best,
Shammon FY

On Thu, Apr 6, 2023 at 9:58 PM neha goyal <ne...@gmail.com> wrote:

> Hello,
> I want to understand the reason behind the different behavior of
> Flink-delta pipelines. When I am running a Kafka in and delta out pipeline
> with fewer resources than needed for a particular pipeline, It fails with
> an OOM error. In the newer version, it takes around 8 minutes to give the
> first exception while in the older version, it fails in 2 minutes only.
> what is the new change which can explain this behavior? we have enabled the
> retrials, hence using the 'first' term.
>
> Pipeline1
> Flink version: 1.16.1
> Delta connector version: 0.6.0
> Time taken for the first OOM error: 8 mins
>
>
> Pipeline2
> Flink version: 1.13.6
> Delta connector version: 0.5.0
> Time taken for the first OOM error: 2 mins
>