You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by deepthi Sridharan <de...@gmail.com> on 2021/04/07 20:04:39 UTC
Compression with rocksdb backed state
I am trying to understand this section on compression of checkpoints which
has me a bit confused
https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression
Could you please confirm if my understanding is correct:
1. Compression is disabled by default for full snapshots and needs to be
turned on if required.
2. Current versions do not support changing the compression type (snappy by
default) when enabled. Future versions will support configuring the
compression type.
3. Compression is enabled by default for incremental snapshot.
4. The compression type cannot be configured for incremental savepoints.
--
Regards,
Deepthi
Re: Compression with rocksdb backed state
Posted by deepthi Sridharan <de...@gmail.com>.
Thank you, that makes sense.
On Thu, Apr 8, 2021 at 12:37 AM Timo Walther <tw...@apache.org> wrote:
> Hi Deepthi,
>
> 1. Correct
> 2. Correct
> 3. Incremental snapshots simply manage references to RocksDB's sstables.
> You can find a full explanation here [1]. Thus, the payload is a
> blackbox for Flink and Flink's compression flag has no impact. So we
> fully rely what RocksDB offers.
> 4. Correct
>
> I hope this helps.
>
> Regards,
> Timo
>
>
> [1]
> https://flink.apache.org/features/2018/01/30/incremental-checkpointing.html
>
>
> On 07.04.21 22:04, deepthi Sridharan wrote:
> > I am trying to understand this section on compression of checkpoints
> > which has me a bit confused
> >
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression
> > <
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression
> >
> >
> > Could you please confirm if my understanding is correct:
> > 1. Compression is disabled by default for full snapshots and needs to be
> > turned on if required.
> > 2. Current versions do not support changing the compression type (snappy
> > by default) when enabled. Future versions will support configuring the
> > compression type.
> > 3. Compression is enabled by default for incremental snapshot.
> > 4. The compression type cannot be configured for incremental savepoints.
> >
> > --
> > Regards,
> > Deepthi
>
>
--
Regards,
Deepthi
Re: Compression with rocksdb backed state
Posted by Timo Walther <tw...@apache.org>.
Hi Deepthi,
1. Correct
2. Correct
3. Incremental snapshots simply manage references to RocksDB's sstables.
You can find a full explanation here [1]. Thus, the payload is a
blackbox for Flink and Flink's compression flag has no impact. So we
fully rely what RocksDB offers.
4. Correct
I hope this helps.
Regards,
Timo
[1]
https://flink.apache.org/features/2018/01/30/incremental-checkpointing.html
On 07.04.21 22:04, deepthi Sridharan wrote:
> I am trying to understand this section on compression of checkpoints
> which has me a bit confused
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression
> <https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression>
>
> Could you please confirm if my understanding is correct:
> 1. Compression is disabled by default for full snapshots and needs to be
> turned on if required.
> 2. Current versions do not support changing the compression type (snappy
> by default) when enabled. Future versions will supportĀ configuring the
> compression type.
> 3. Compression is enabled by default for incremental snapshot.
> 4. The compression type cannotĀ be configured for incremental savepoints.
>
> --
> Regards,
> Deepthi