checkpoint _metadata file has >20x different in size among different check-points

[Yu Yang <yu...@gmail.com>]

Hi all,

We have a flink job that does check-pointing per 10 minutes. We noticed
that for the check-points of this job,  the _metadata file size can vary a
lot. In some checkpoint, we observe that _metadata file size was >900MB,
while in some other check-points of the same job, the _metadata file size
is < 4MB.  Any insights on what may cause the difference?

Thank you!

Regards,
-Yu

Re: checkpoint _metadata file has >20x different in size among different check-points

[Congxian Qiu <qc...@gmail.com>]

Hi

Maybe there contains some ByteStreamStateHandle in the checkpoint, if you
want to verify this, maybe you can configure
`state.backend.fs.memory-threshold` to verify it. Please be careful to set
this config, because it may produce many files with small size.

Best,
Congxian


Arvid Heise <ar...@ververica.com> 于2020年3月5日周四 上午2:26写道：

> Hi Yu,
>
> are you using incremental checkpoints [1]? If so, then the smaller
> checkpoints would be the deltas and the larger the complete state.
>
> [1]
> https://flink.apache.org/features/2018/01/30/incremental-checkpointing.html
>
> On Wed, Mar 4, 2020 at 6:41 PM Yu Yang <yu...@gmail.com> wrote:
>
>> Hi all,
>>
>> We have a flink job that does check-pointing per 10 minutes. We noticed
>> that for the check-points of this job,  the _metadata file size can vary a
>> lot. In some checkpoint, we observe that _metadata file size was >900MB,
>> while in some other check-points of the same job, the _metadata file size
>> is < 4MB.  Any insights on what may cause the difference?
>>
>> Thank you!
>>
>> Regards,
>> -Yu
>>
>

Re: checkpoint _metadata file has >20x different in size among different check-points

[Arvid Heise <ar...@ververica.com>]

Hi Yu,

are you using incremental checkpoints [1]? If so, then the smaller
checkpoints would be the deltas and the larger the complete state.

[1]
https://flink.apache.org/features/2018/01/30/incremental-checkpointing.html

On Wed, Mar 4, 2020 at 6:41 PM Yu Yang <yu...@gmail.com> wrote:

> Hi all,
>
> We have a flink job that does check-pointing per 10 minutes. We noticed
> that for the check-points of this job,  the _metadata file size can vary a
> lot. In some checkpoint, we observe that _metadata file size was >900MB,
> while in some other check-points of the same job, the _metadata file size
> is < 4MB.  Any insights on what may cause the difference?
>
> Thank you!
>
> Regards,
> -Yu
>