You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Gabor Somogyi <ga...@gmail.com> on 2024/10/15 14:26:16 UTC

Major performance degradation in case of list operator state restore from S3

Hi All,

As a result of our investigation we've found major performance degradation
described here [1].

TLDR: list operator state restore from S3 is significantly slower from
1.18.2. Both uncompressed
and compressed state is effected. ~6Mb restore takes ~62 hours.

We've created already a solution for uncompressed state [2]. This works
fine under heavy load,
but important to mention here that the compressed state still suffers from
this issue.

Since I was not there when the compressed state was born I would ask some
help to sort that part
out too.

[1] https://issues.apache.org/jira/browse/FLINK-36530
[2] https://github.com/apache/flink/pull/25509

BR,
G

Re: Major performance degradation in case of list operator state restore from S3

Posted by Gabor Somogyi <ga...@gmail.com>.
Just for the record, this conversation continued and resolved in user list
[1] and in the PR [2].

[1] https://lists.apache.org/thread/h5mv6ld4l2g4hsjszfdos9f365nh7ctf
[2] https://github.com/apache/flink/pull/25509

G


On Tue, Oct 15, 2024 at 4:26 PM Gabor Somogyi <ga...@gmail.com>
wrote:

> Hi All,
>
> As a result of our investigation we've found major performance degradation
> described here [1].
>
> TLDR: list operator state restore from S3 is significantly slower from
> 1.18.2. Both uncompressed
> and compressed state is effected. ~6Mb restore takes ~62 hours.
>
> We've created already a solution for uncompressed state [2]. This works
> fine under heavy load,
> but important to mention here that the compressed state still suffers from
> this issue.
>
> Since I was not there when the compressed state was born I would ask some
> help to sort that part
> out too.
>
> [1] https://issues.apache.org/jira/browse/FLINK-36530
> [2] https://github.com/apache/flink/pull/25509
>
> BR,
> G
>
>