You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Piotr Nowojski (Jira)" <ji...@apache.org> on 2020/05/28 06:16:00 UTC
[jira] [Closed] (FLINK-17928) Incorrect state size reported when
using unaligned checkpoints
[ https://issues.apache.org/jira/browse/FLINK-17928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Piotr Nowojski closed FLINK-17928.
----------------------------------
Resolution: Fixed
merged to release-1.11 as a9782a2483
merged to master as f89c606f12
> Incorrect state size reported when using unaligned checkpoints
> ---------------------------------------------------------------
>
> Key: FLINK-17928
> URL: https://issues.apache.org/jira/browse/FLINK-17928
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.11.0
> Reporter: Piotr Nowojski
> Assignee: Roman Khachatryan
> Priority: Critical
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Even when checkpoints on HDFS are between 100-300MBs, the reported state size is in orders of magnitude larger with values like:
> {noformat}
> 1GiB 1.5TiB 2.0TiB 2.1TiB 2.1TiB
> 148GiB 148GiB 148GiB 148GiB 148GiB 148GiB
> {noformat}
> it's probably because we have multiple {{Collection<InputChannelStateHandle>}}, and each of the individual handle returns the same value from {{AbstractChannelStateHandle#getStateSize}} - the full size of the spilled data, ignoring that only small portion of those data belong to a single input channel/result subpartition. In other words {{
> org.apache.flink.runtime.state.AbstractChannelStateHandle#getStateSize}} should be taking the offsets into account and return only the size of the data that belong exclusively to this handle.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)