[jira] [Created] (FLINK-17820) Memory threshold is ignored for channel state

["Roman Khachatryan (Jira)" <ji...@apache.org>]

Roman Khachatryan created FLINK-17820:
-----------------------------------------

             Summary: Memory threshold is ignored for channel state
                 Key: FLINK-17820
                 URL: https://issues.apache.org/jira/browse/FLINK-17820
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Checkpointing, Runtime / Task
    Affects Versions: 1.11.0
            Reporter: Roman Khachatryan
            Assignee: Roman Khachatryan
             Fix For: 1.11.0


Config parameter state.backend.fs.memory-threshold is ignored for channel state. Causing each subtask to have a file per checkpoint. Regardless of the size of channel state (of this subtask).

This also causes slow cleanup and delays the next checkpoint.

 

The problem is that {{ChannelStateCheckpointWriter.finishWriteAndResult}} calls flush(); which actually flushes the data on disk.

 

From FSDataOutputStream.flush Javadoc:

A completed flush does not mean that the data is necessarily persistent. Data persistence can is only assumed after calls to close() or sync().

 

Possible solutions:

1. not to flush in {{ChannelStateCheckpointWriter.finishWriteAndResult (which can lead to data loss in a wrapping stream).}}

{{2. change }}{{FsCheckpointStateOutputStream.flush behavior}}

{{3. wrap }}{{FsCheckpointStateOutputStream to prevent flush}}{{}}{{}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)