[jira] [Closed] (FLINK-30189) HsSubpartitionFileReader may load data that has been consumed from memory

["Xintong Song (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-30189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xintong Song closed FLINK-30189.
--------------------------------
    Fix Version/s: 1.17.0
                   1.16.1
       Resolution: Fixed

- master (1.17): 4c67f8fca529a72389d69990307bbf78fcd3d99d
- release-1.16: a9e65bc2377ee7a4b3599b58a58ff0301b79c5d8

> HsSubpartitionFileReader may load data that has been consumed from memory
> -------------------------------------------------------------------------
>
>                 Key: FLINK-30189
>                 URL: https://issues.apache.org/jira/browse/FLINK-30189
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Network
>    Affects Versions: 1.16.0, 1.17.0
>            Reporter: Weijie Guo
>            Assignee: Weijie Guo
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.17.0, 1.16.1
>
>
> In order to solve the problem that data cannot be read from the disk correctly after failover, we changed the calculation logical of the buffer's readable state in FLINK-29238.  Buffers that are greater than consumingOffset and have been released can be pre-load from file. However, the update of consumingOffset is asynchronous, If it lags behind the actual consumption progress, the buffer will have a chance to be load from the disk again. 
> IMO, we can record the consumed status of buffer by each consumer in the InternalRegion. Only the buffers that have not been consumed and have been released will be considered as readable. In the case of failover, a new consumerId will be generated, so all buffers will be considered as unconsumed and can be correctly read from the disk too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)