You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Gabor Somogyi <ga...@gmail.com> on 2024/10/15 14:26:16 UTC
Major performance degradation in case of list operator state restore from S3
Hi All,
As a result of our investigation we've found major performance degradation
described here [1].
TLDR: list operator state restore from S3 is significantly slower from
1.18.2. Both uncompressed
and compressed state is effected. ~6Mb restore takes ~62 hours.
We've created already a solution for uncompressed state [2]. This works
fine under heavy load,
but important to mention here that the compressed state still suffers from
this issue.
Since I was not there when the compressed state was born I would ask some
help to sort that part
out too.
[1] https://issues.apache.org/jira/browse/FLINK-36530
[2] https://github.com/apache/flink/pull/25509
BR,
G
Re: Major performance degradation in case of list operator state restore from S3
Posted by Gabor Somogyi <ga...@gmail.com>.
Just for the record, this conversation continued and resolved in user list
[1] and in the PR [2].
[1] https://lists.apache.org/thread/h5mv6ld4l2g4hsjszfdos9f365nh7ctf
[2] https://github.com/apache/flink/pull/25509
G
On Tue, Oct 15, 2024 at 4:26 PM Gabor Somogyi <ga...@gmail.com>
wrote:
> Hi All,
>
> As a result of our investigation we've found major performance degradation
> described here [1].
>
> TLDR: list operator state restore from S3 is significantly slower from
> 1.18.2. Both uncompressed
> and compressed state is effected. ~6Mb restore takes ~62 hours.
>
> We've created already a solution for uncompressed state [2]. This works
> fine under heavy load,
> but important to mention here that the compressed state still suffers from
> this issue.
>
> Since I was not there when the compressed state was born I would ask some
> help to sort that part
> out too.
>
> [1] https://issues.apache.org/jira/browse/FLINK-36530
> [2] https://github.com/apache/flink/pull/25509
>
> BR,
> G
>
>